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Executive  Summary 


Systems  Engineering  Research  Center  (SERC)  Research  Topic  34  (RT-34) 
Expedited  Systems  Engineering  for  Rapid  Capability  and  Urgent  Needs 

Stevens  Institute  of  Technology 
Air  Force  Institute  of  Technology 
University  of  Southern  California 


Introduction 

This  Systems  Engineering  Research  Center  (SERC)  research  topic  34  (RT-34)  examines  expedited  systems 
engineering  (SE)  as  applied  to  rapid  capability  and  urgent  needs  as  developed  in  response  to  changing  threats. 

The  Defense  Science  Board  (DSB)  Task  Force  on  the  Fulfillment  of  Urgent  Operational  Needs  (July  2009)  identified 
more  than  20  rapid-reaction  programs  and  organizations  addressing  DoD  urgent  warfighter  needs.  Later, 
OSD/DDR&E  conducted  a  study  called  "Rapid  Capability  Fielding  Toolbox"  released  in  March  2010.  Recently  GAO 
has  examined  how  well  the  DoD  handles  rapid  solutions  to  Joint  Urgent  Needs,  and  they  have  also  specifically 
examined  the  acquisition  activities  for  Special  Ops  urgent  needs.  These  studies  determined  that  the  standard 
Department  of  Defense  acquisition  process  is  not  designed  to  respond  to  the  dynamic  environment  of  rapid 
acquisition.  These  reports  and  others  have  also  documented  the  problems  and  possible  solutions  surrounding 
rapid  acquisition,  rapid  fielding  and/or  rapid  prototyping.  The  recommendations  have  included  acquisition 
process  changes  and  introduction  of  new  SE  tools. 

Lifecycle  of  urgent  needs  programs  is  driven  by  "time  to  market"  as  opposed  to  complete  satisfaction  of  static 
requirements,  with  delivery  expected  in  days/months  versus  years/decades.  The  processes  and  practices  applied 
to  urgent  needs  must  add  value  and  not  require  an  excessive  bureaucratic  oversight  to  implement,  while  at  the 
same  time  address,  understand,  and  manage  risk  such  that  programs  can  understand  better  where  to  include, 
truncate,  or  eliminate  systems  engineering  (SE)  practices  and  processes.  The  original  RT-34  hypothesis  was  that 
by  defining,  identifying,  testing,  and  ultimately  implementing  expedited  SE  processes  and  practices,  capability  that 
results  from  urgent  needs  may  be  more  effective,  efficient,  and  longer  lasting  in  the  field.  Potential  second  order 
effects  are  that  expedited  SE  as  applied  to  urgent  needs  could  streamline  specific  future  SE  practices,  as  urgent 
becomes  "normal,"  and  findings  could  eventually  improve  SE  processes  for  traditional  programs  as  well.  The 
intersection  of  these  two  extremes  could  become  a  "hybrid"  approach  to  SE. 

The  RT-34  research  team  set  out  with  the  goal  to  identify  the  factors  that  contribute  positively  to  rapid  acquisition 
focusing  on  the  systems  engineering  process.  One  can  hypothesize  that  certain  critical  success  factors  from  those 
organizations  that  do  rapid  acquisition  may  well  be  transferrable  to  traditional  acquisition.  These  critical  success 
factors  may  include  aspects  from  their  personnel/organization,  the  processes  they  use,  or  how  they  approach  the 
product/system/solution. 

This  research  effort  studied  over  30  organizations  and  individuals  -  from  commercial,  civil  and  DoD  sectors  - 
which  have  known  experience  in  rapid  development  as  shown  in  Table  1.  These  organizations  and  individuals 
were  at  the  headquarters  level  and  often  represented  a  portfolio  of  rapid  programs.  The  investigation  attempted 
to  identify  any  people,  process  or  product  related  behaviors  that  the  organizations  thought  most  contributed  to 
expedited  systems  engineering.  The  site  visit  discussions  showed  that  expedited  SE  was  conducted  in  the  context 
of  overall  rapid  development,  and  the  responses  from  individuals  focused  more  on  the  contributions  of  people  to 
rapid  development  and  acquisition  success  instead  of  a  "secret  sauce"  of  how  to  expedite  SE. 

It  is  also  necessary  to  define  what  we  mean  by  "rapid."  Most  define  "rapid"  as  generally  delivering  a  capability  as 
quickly  as  2  months  and  no  longer  than  24  months.  However,  we  have  also  seen  "rapid"  referred  to  as  "half  the 
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time  of  traditional  acquisition."  The  definition  must  also  consider  the  context  of  the  acquisition  environment  as 
well  as  the  characteristics  of  the  fielded  capability,  such  as  amount  of  operational  capability/performance, 
warfighter  satisfaction,  safety  of  operations,  manufacturability,  and  sustainability  (repairs,  parts,  etc.).  Many 
other  design  considerations  may  be  critical,  which  must  be  balanced  with  time  to  market.  We  refer  to  "expedited 
systems  engineering"  as  the  set  of  systems  engineering  and  engineering  management  activities  used  during  a 
rapid  acquisition,  which  may  be  tailored  and  scaled  appropriately  (or  unfortunately  eliminated  or  delayed 
inappropriately).  This  last  approach  has  been  termed  technical  debt. 

The  RT-34  team  found  an  enthusiasm  that  is  infectious  throughout  the  organizations  that  regularly  practice 
"rapid"  successfully.  This  finding  revealed  that  rapid  organizations  have  a  certain  culture  that  fosters  and  enables 
a  certain  esprit  de  corps.  We  hope  that  exhilaration  can  spread  throughout  the  entire  acquisition  community. 


Methodology  Overview 

Research  on  success  factors  and  factors  categories  is  well  documented  for  varied  purposes  in  the  Industrial 
Engineering  community.  Various  types  of  success  have  been  documented  in  project  planning,  control  and 
monitoring,  project  management,  organizations,  external  environment,  process  definition,  and  technical 
considerations.  A  recent  research  paper  on  "Critical  Success  Factors  for  Rapid,  Innovative  Solutions,"  (Lane  et  al) 
concluded  that  successful  innovative  organizations  share  certain  characteristics  regarding  culture,  risk,  team  size, 
solution  patterns,  engineering  processes  and  innovation.  These  findings  parallel  Kelly  Johnson's  famous  "14  Rules 
of  Management"  from  the  Lockheed  Skunk  Works  program,  where  his  motto  was,  "Be  quick,  be  quiet,  and  be  on 
time." 

Based  on  these  initial  factors,  literature  review,  inquiry  and  input  from  the  SERC  research  council  and  the 
researchers'  personal  experience,  a  list  of  guiding  questions  was  identified  for  unstructured  site  visits  with  subject 
matter  experts  (SME)  at  organizations  known  for  conducting  rapid  development  and/or  rapid  acquisition.  It  was 
assumed  that  such  organizations  also  practiced,  or  would  be  familiar  with,  expedited  systems  engineering  in  the 
context  of  rapid  development  and  fielding  of  systems.  The  questions,  along  with  a  short  description  of  the 
research,  were  provided  to  each  organization  ahead  of  the  visit. 

After  the  first  week  of  research,  it  was  clear  that  the  answers  were  naturally  grouped  into  three  categories  as 
follows.  A  revised  list  of  37  questions  was  clustered  accordingly  and  used  for  the  remaining  period  of  data 
collection: 

1.  People  /  organizational  questions 

2.  Process  questions  regarding  Systems  Engineering  and  technical  engineering  management 

3.  Product  questions  regarding  architectural  aspects  of  the  solution. 

It  should  be  noted  that  the  questions  were  very  engineering  focused,  to  examine  the  systems  engineering 
technical  and  technical  management  processes  (e.g.,  risk,  requirements,  technical  decision  making,  modeling, 
documentation,  etc)  as  well  as  the  product  or  solution  architecture  (e.g.,  interfaces,  technical  maturity, 
complexity,  etc).  The  responses,  however,  nearly  always  pointed  back  to  the  people,  who  had  a  set  of  career 
experiences  and  a  way  of  thinking  that  enabled  them  to  make  appropriate  judgment  calls  on  what  systems 
engineering  was  needed,  and  that  this  process  was  enabled  through  the  culture  and  leadership  of  the 
organization.  The  responses  could  be  interpreted  by  the  researchers,  and  the  reader,  to  better  describe  system 
development  or  system  acquisition,  than  strictly  systems  engineering. 

The  team  targeted  those  organizations  who  have  been  acquiring  Joint  Urgent  Operational  Need  Statement 
(JUONS)  solutions  or  who  had  expertise  in  aspects  of  rapid  non-traditional  acquisition.  Predominantly,  the 
organizations  were  either  Government  defense  acquisition  offices  or  select  defense  industries.  These 
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organizations  are  known  for  rapid,  non-traditional  acquisition.  A  short  planning  phase  identified  organizations 
practicing  expedited  systems  engineering  such  as  the  Prototype  Integration  Facility  at  the  U.S.  Army  Aviation  and 
Missile  Research  Development  and  Engineering  Center  in  Huntsville,  AL,  and  the  U.S.  Special  Operations 
Command  (SOCOM)  at  McDill  AFB,  FL.  The  organizations  were  identified  by  those  known  to  the  SERC  sponsors, 
the  SERC  Research  Council,  and  the  researchers;  those  identified  in  the  DSB  and  GAO  reports;  and  via  referrals 
from  other  rapid  organizations. 

Initially  our  list  included  a  variety  of  organizations  from  government  and  industry,  as  the  original  research  plan 
called  for  us  to  analyze  the  start  of  the  art  in  expedited  SE  within  the  DoD  and  commercial  sector.  As  time  went 
on,  we  narrowed  the  focus  to  government  rapid  acquisition  offices,  as  these  more  closely  represented  the  user 
community  of  interest  to  the  sponsor  and  were  more  consistent  with  the  future  research  intent  to  validate  an 
expedited  SE  framework  on  a  DoD  acquisition  program. 

The  notes  from  over  30  discussions  with  organizations  and  individuals  were  coded  and  tagged  based  on  patterns 
of  consistent  responses.  The  full  list  of  site  visits  is  shown  in  Table  1,  which  reflects  25  locations,  some  with 
discussions  with  multiple  SMEs.  The  list  is  in  chronological  order,  showing  the  increased  focus  on  defense  and  Air 
Force  organizations  as  the  research  progressed. 

Table  1:  List  of  Site  Visits  Conducted 


1 

The  Aerospace  Corporation  Concept  Design  Center  (CDC),  Los  Angeles,  CA 

2 

American  Institute  of  Aeronautics  and  Astronautics  (AIAA)  Space  2011  conference,  Long  Beach,  CA  (various  discussions 
with  SMEs  from  government,  industry,  and  academia) 

3 

An  aerospace  industry  innovation  lab,  Southern  California 

4 

A  rocket  engine  design  company.  Southern  California 

5 

Annual  SERC  Research  Review  (ASRR),  College  Park,  MD  (various  discussions  and  presentation  notes,  including  a  panel 
discussion  on  this  research  subject) 

6 

Small  satellite  development  company 

7 

NASA  Goddard  Space  Flight  Center  (GSFC),  Greenbelt,  MD  (multiple  separate  discussions) 

8 

Operationally  Responsive  Space  (ORS)  Office,  Kirtland  AFB,  NM  (multiple  discussions) 

9 

USAF  Space  Development  and  Test  Directorate,  Kirtland  AFB,  NM  (multiple  discussions) 

10 

Int'l  Symposium  for  Personal  &  Commercial  Spaceflight  (ISPCS)  (several  discussions  w/speakers  &  attendees) 

11 

International  Women's  Forum  Annual  Conference,  Washington,  DC  (notes  from  panel  presentation) 

12 

Air  Force  Rapid  Capabilities  Office  (RCO),  Washington  DC 

13 

Intelligence  Community,  Washington  DC  (multiple  organizations  represented) 

14 

Georgia  Tech  Research  Institute  (GTRI)  Collaborative  Visualization  Environment  ("COVE"),  Atlanta,  GA 

15 

U.S.  Army  Prototype  Integration  Facility  (PIF),  Huntsville,  AL 

16 

Aerospace  /  Engineering  small  business,  Huntsville,  AL  (multiple  discussions) 

17 

Big  Safari  -  Program  office  and  AF  Program  Executive  Office,  ISR-SOF,  Wright-Patterson  AFB  OH 

18 

Air  Force  Research  Lab  (AFRL)  Center  for  Rapid  Product  Development  (CRPD),  Wright-Patterson  AFB,  OH 

19 

AF  Aeronautical  Systems  Center,  Rapid  Development  Integration  Facility  (RDIF),  Wright-Patterson  AFB,  OH 

20 

AFRL  Air  Vehicles  Directorate,  Wright-Patterson  AFB,  OH  (multiple  discussions) 

21 

Air  Force  Space  Command  /  A5,  Peterson  AFB,  CO 

22 

Air  Force  Space  and  Missile  Systems  Center,  Rapid  Reaction  Branch,  Colorado  Springs,  CO 

23 

Air  Force  Tactical  Exploitation  of  National  Capabilities  (TENCAP),  Schriever  AFB,  CO 

24 

Air  Force  Space  and  Missile  Systems  Center  HQ,  Los  Angeles,  CA 

25 

SOCOM  Research  and  Development  Acquisition  Center  (SORDAC),  MacDill  AFB,  FL 
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In  total,  the  25  site  visits  represented: 

•  15  government  defense  and  intelligence  community,  of  which  10  were  Air  Force 

•  4  industry  (all  4  conduct  work  for  defense,  civil,  and  commercial  customers) 

•  1  government  civil  space 

•  1  Federally  Funded  Research  and  Development  Center  (FFRDC) 

•  1  educational  institution 

•  with  the  remaining  3  "site  visits"  conducted  at  conferences  incorporating  multiple  individuals  from  all  of 
these  sectors. 


The  method  used  in  this  research  is  based  on  grounded  theory.  Grounded  theory  is  a  type  of  qualitative  research 
methodology  that  allows  theories  to  emerge  from  collected  data.  This  collection  of  data  comes  from  notes  during 
discussions  with  the  leadership  of  the  "rapid"  organizations— essentially  experts  in  the  field— to  discern  what 
made  them  successful  and  discover  what  drove  their  processes.  The  research  follows  a  systematic,  yet  flexible 
process  to  collect  data,  code  and  analyze  the  data,  make  connections,  and  see  what  theories  can  be  generated. 
This  "open  coding"  of  labels  is  an  important  part  of  the  analysis  concerned  with  identifying,  naming  or  labelling, 
categorizing,  and  describing  phenomena  found  in  the  discussion  notes.  In  this  case,  the  theory  is  a  set  of 
principles  for  successful  DoD  rapid  development,  with  a  focus  on  systems  engineering. 


Summary  of  Findings 

It  was  clear  from  the  site  visits  that  expedited  systems  engineering  is  conducted  in  a  context  of  rapid  development 
and  acquisition.  When  SMEs  talk  about  what  their  organizations  do,  they  focus  on  the  bigger  context,  even 
though  our  questions  focused  on  systems  engineering.  The  responses  showed  that  rapid  development  requires 
an  integrated  approach:  People  making  judgments,  Processes  for  task  reductions,  and  Product  aspects  focused  on 
rapid  objectives.  The  "3  P's"  of  "people,  process,  and  product"  show  organizations  mindful  of  their  people  (with 
desired  expertise,  trust,  motivation  and  expectations),  the  requirements,  and  the  use  of  iteration  to  field  solutions 
quickly  and  refine  those  requirements.  Rapid  organizations  also  seek  out  flexibility  in  process,  contracting  and 
right-sizing  the  solution  to  avoid  undue  oversight.  Further  analysis  reflects  findings  pertaining  to  efficient 
information/knowledge  sharing,  generally,  but  not  exclusively  achieved  by  small  team  collocation.  These 
organization  were  mindful  of  risk,  but  not  to  its  complete  removal  from  a  program  (if  ever  possible).  Rapid 
projects  demonstrated  both  an  exploitation  of  mature  technologies  and  rapid  solutions  during  their  planning 
function,  as  well  as  the  efficient  and  effective  ability  to  execute  those  plans. 

Like  the  different  DoD  Acquisition  Category  (ACAT)  levels  defined  by  RDT&E  and  Production  dollar  amounts,  four 
different  lanes  of  acquisition  were  identified  that  represent  the  different  types  of  solutions  selected  by  these 
rapid  organizations.  These  four  lanes  consisted  of:  laboratory  or  operational  prototyping,  platform  engineering 
and  modification,  integrated  solutions  and  traditional  weapon  system  acquisition.  Huge  flexibility  exists  within 
and  throughout  these  lanes. 

While  we  expected  to  find  explanations  for  rapid  SE  processes  practiced  by  rapid  organizations,  the  number  one 
finding  was  the  strong  role  people  play  in  a  rapid  context  and  an  overwhelming  view  on  the  importance  of 
"designing  the  design  team."  This  is  reflected  in  the  culture  and  organizational  ethos  of  rapid  organizations. 
People  create  and  implement  the  process  to  make  the  product.  We  also  observed  that  the  rapid  acquisition 
workforce  is  exposed  to  "hands-on"  full  lifecycle  decision-making  and  their  consequences.  These  experiences 
were  often  cited  as  critical  rationale  for  someone  being  selected  (or  self-selecting)  for  a  rapid  project  or  career. 
These  experiences  are  in  contrast  to  what  is  experienced  in  a  traditional  full  DoD5000  acquisition  cycle. 

For  example,  in  the  18  months  it  might  take  to  field  a  rapid  program,  a  traditional  acquisition  would  see  a  portion 


Contract  Number:  H98230-08-D-0171 


Page  5 

Report  No.  SERC-2012-1R-034 


TO  0015,  RT034 


UNCLASSIFIED 


of  one  milestone.  The  compressed  nature  of  rapid  acquisition  means  that  people  have  a  greater  appreciation  for 
the  end  objective  of  the  product  -  i.e.,  to  quickly  deliver  the  end  effect  required  by  the  warfighter  in  the  field. 
While  these  observations  may  not  seem  new,  they  provide  the  foundation  for  a  broader  framework  of  rapid 
development. 

It  must  be  pointed  out  that  the  site  visit  discussions  generally  took  place  at  headquarters  level  and  did  not  focus 
on  particular  projects  or  programs.  The  observations  captured  may  be  a  result  of  bias  in  talking  to  higher 
management  (across  programs)  rather  than  lower-level  management,  or  engineering,  within  a  small  rapid  project. 

An  original  goal  of  the  RT-34  research  was  to  develop  a  framework  for  scaling  SE  activities  in  response  to  different 
development  constraints,  such  as  reduced  development  time,  with  intent  to  prepare  a  plan  to  validate  the 
framework  on  a  DoD  acquisition  program.  As  the  research  evolved,  we  discovered  that  in  order  to  tailor  or  scale 

SE  process,  the  right  people  with  the  right  perspective  and  experiences  needed  to  be  in  place,  in  the  environment 

that  enabled  them  to  be  successful.  We  identified  several  ways  to  validate  the  framework  as  well  as  future 
research  questions  that  could  be  explored  in  the  meantime. 

Based  on  observation,  interviews,  and  literature,  a  series  of  observations,  or  principles,  begins  to  emerge  that 
reflects  a  framework  of  rapid  development.  While  we  originally  proposed  a  stacked  model  with  increasing  rigor, 
we  finally  elected  to  not  infer  any  precedence,  hierarchy,  or  increasing  systems  engineering  rigor.  For 
presentation  in  this  report,  we  break  our  findings  into  three  groups: 

•  Group  1:  Direct  Responses, 

•  Group  2:  Direct  Observations,  and 

•  Group  3:  Inferred  Organizational  Characteristics. 

The  Direct  Responses  in  Group  1  provide  an  efficient  affinity  across  all  answers  to  our  37  questions.  This  set 
represents  the  top  11  "Organizational  Best  Practices".  We  observed  strong  evidence  across  the  site  visits  for  all 
11  observations.  In  the  process  of  validating  the  responses  with  the  SMEs,  reviewing  against  literature,  and 
socializing  the  observations  with  others  in  the  rapid  and  traditional  community,  we  realized  that  the  11 
observations  are  "not  new"  and  also  do  not  appear  to  be  unique  to  rapid  programs.  We  characterize  these  as 
practices  that  can  be  found  in  both  rapid  and  traditional  development  programs. 

Group  2  "Flexible  Acquisition  Practices"  reflects  Direct  Observations  that  emerged  from  the  site  visits,  but  were 
not  direct  responses  to  the  prepared  questions.  This  group  collects  observations  based  on  flexibility  in  acquisition 
practices,  whether  it  be  contract  type,  finance  type,  program  management  approach,  documentation  required, 
level  of  insight/oversight,  etc.  These  practices  are  seen  as  critical  for  a  rapid  business  environment.  We  originally 
did  not  seek  out  different  types  of  rapid  categories,  or  strictly  acquisition  management  (such  as  contracting) 
findings;  in  fact,  a  ground  rule  of  the  research  was  that  we  would  not  seek  to  change  the  existing  DoD5000 
process.  Flowever,  these  observations  about  the  acquisition  environment  emerged  from  the  site  visits  and  could 
not  be  ignored. 

Group  3,  the  set  of  "Inferred  Organizational  Characteristics,"  represents  aspects  of  a  "Go  Fast"  Culture  seen  in  the 
rapid  organizations.  This  is  where  rapid  organizations  start  to  differ  from  traditional  ones,  and  there  is  a  shift  in 
energy,  commitment,  and  knowledge.  These  findings  are  motivated  by  an  analysis  of  effective  enterprise  culture 
and  business  agility  literature.  The  Group  3  characteristics  include: 
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•  Rapid  organizations  use  intense  and  efficient  knowledge-sharing  to  enable  stabilization  and 
synchronization  of  information 

•  Rapid  organizations  are  characterized  by  a  risk-tolerant  culture 

•  Rapid  organizations  are  structured  for  ambidexterity  with  a  balance  between  exploration  and  exploitation 

•  Rapid  development  organizations  exercise  Real-time  Management,  with  both  empowerment  at  the 
lowest  level  at  which  sufficient  knowledge  and  experience  exists  to  make  appropriate  decisions  as  well  as 
rapid  access  to  top-level  leadership  to  facilitate  decision  velocity. 

Real-time  management  represents  an  integrative  characteristic;  it  requires  intense  knowledge  sharing,  risk- 
focused  culture  (people  knowing  what  risks  to  take  and  when  and  how),  and  the  organizational  ambidexterity  that 
comes  from  a  balance  of  exploration  and  exploitation. 

The  resulting  RT-34  Expedited  SE  Framework  is  captured  in  Figure  1. 
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Figure  1:  RT-34  Expedited  SE  Framework 

This  report  goes  into  more  detail  about  specific  findings  and  recommendations  associated  with  each  level  and 
several  potential  follow-on  research  thrusts.  Two  main  research  questions  emerged  that  resonate  with  the 
sponsors: 

1.  What  are  the  principles  and  attributes  that  are  seen  in  each  level  of  framework  and  do  they  apply  equally 
across  all  domains  and  all  lanes  of  acquisition? 

2.  Flow  can  human  capital  development  improve  by  embedding  acquisition  personnel  in  programs  to 
develop  these  principles  and  attributes? 
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Other  questions  arose  after  a  peer-review  "deep  dive"  of  the  RT-34  project.  These  questions  included: 

1.  Is  there  a  depiction  of  the  framework  that  is  hierarchical  in  nature,  such  that  it  infers,  and  justifies, 
precedence  or  causality? 

2.  How  is  DoD  rapid  acquisition  different  from  DoD  non-rapid  or  non-DoD  rapid  acquisition? 

3.  How  can  a  traditional  acquisition  (large  ACAT  1)  embrace  appropriate  factors  of  rapid  organizations, 
effectively  becoming  a  "hybrid"  program? 

The  answers  to  these  questions  will  be  referred  to  any  follow-on  research. 
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1  Introduction 


1.1  Motivation 

This  SERC  Research  Task  34  (RT-34)  set  out  to  examine  expedited  systems  engineering  (SE)  as  applied  to  rapid 
capability  and  urgent  needs  as  developed  in  response  to  changing  threats.  The  lifecycle  of  urgent  needs  programs 
is  driven  by  "time  to  market"  as  opposed  to  complete  satisfaction  of  static  requirements,  with  delivery  expected 
in  days/months  versus  years/decades.  The  processes  and  practices  applied  to  urgent  needs  programs  must  add 
value  and  not  require  an  excessive  bureaucratic  oversight  to  implement,  while  at  the  same  time  address, 
understand,  and  manage  risk  such  that  program  management  can  understand  better  where  to  include,  truncate, 
or  eliminate  systems  engineering  practices  and  processes.  The  original  hypothesis  was  that  by  defining, 
identifying,  testing,  and  ultimately  implementing  expedited  SE  processes  and  practices,  capability  that  results 
from  urgent  needs  may  be  more  effective,  efficient,  and  longer  lasting  in  the  field.  Potential  second  order  effects 
are  that  expedited  SE  as  applied  to  urgent  needs  programs  could  streamline  specific  future  SE  practices,  as  urgent 
becomes  "normal,"  and  findings  could  eventually  improve  SE  processes  for  traditional  programs  as  well.  The 
research  evolved,  through  data  collected  at  site  visits  with  subject  matter  experts  (SMEs),  to  examining  the 
importance  of  the  overall  context  of  rapid  development,  i.e.,  the  environment  in  which  expedited  SE  is  conducted. 

This  research  is  timely  as  the  2011  Defense  Authorization  Act  called  for  the  establishment  of  a  "joint  urgent 
operational  need  response  fund"  as  part  of  its  2012  budget  request.  Further,  the  Air  Force  announced  that  it 
would  manage  its  Next  Generation  Bomber  under  the  auspices  of  the  Rapid  Capabilities  Office  because  it  offers 
more  streamlined  acquisition  than  other  channels.  RT-34  was  proposed  to  focus  on  leveraging  currently  available 
methods,  processes  and  tools  to  create  a  coherent  expedited  SE  strategy  that  can  be  validated  in  practice. 

Typical  DoD  systems  experience  change  on  many  different  time  scales.  To  remain  effective,  these  systems  must 
accommodate  change  on  each  of  these  scales.  As  was  noted  in  the  SERC  Systems  2020  report  (Boehm  et  al., 

2010),  systems  engineering  capabilities  need  to  be  developed  that  achieve  dramatic  reduction  in  time  to:  develop 
a  fieldable  first-article  product,  implement  foreseeable  classes  of  fielded  systems  changes,  and  rapidly  adapt  to 
unforeseeable  threats.  Existing  systems  engineering  tools,  processes,  and  technologies  poorly  support  rapid 
design  changes  or  capability  enhancements  within  acceptable  cost  and  schedule  constraints.  Perhaps  an  artifact 
of  the  DoD  acquisition  process,  rapid  development  has  achieved  point  solutions,  which  make  adaptation 
cumbersome  and  impractical  in  theatre.  To  increase  development  efficiency  and  ensure  flexible  solutions  in  the 
field,  systems  engineers  need  powerful,  agile,  interoperable,  and  scalable  processes,  tools  and  techniques.  One  of 
the  DoD's  goals  is  to  embrace  commercial  marketplace  approaches  so  that  warfighters  can  have  innovative 
solutions  that  are  transitioned  into  capabilities  in  months,  not  years. 

This  research  also  has  roots  in  the  "Rapid  Capability  Fielding  Toolbox"  60-day  study  conducted  by  DDR&E  and 
released  in  March  2010.  This  study  noted  that  the  standard  Department  of  Defense  acquisition  process  is  not 
designed  to  respond  to  the  dynamic  environment  of  rapid  needs,  and  called  for  further  investigations  into 
concept  engineering,  capability  engineering,  and  cultural  change.  Many  DoD  rapid  development  cells  were 
identified,  and  a  number  of  interviews  with  individuals  working  on  rapid  development  projects  were  conducted. 

Several  Defense  reports  have  documented  in-depth  studies  on  the  problems  and  possible  solutions  surrounding 
rapid  acquisition,  rapid  fielding  and/or  rapid  prototyping.  (Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  2011)  (Office  of  Secretary  of  Defense,  Deputy  Director  for  Research  & 
Engineering,  March  2010)  (Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics, 
2009).  Recommendations  include  acquisition  process  changes  and  introduction  of  new  SE  tools  designed  to 
handle  a  rapidly  changing  environment  that  challenges  the  current  acquisition  community. 
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First,  most  define  "rapid"  as  generally  delivering  a  capability  as  quickly  as  2  months  and  no  longer  than  24  months 
(DSB,  2009).  However  we  have  also  seen  "rapid"  referred  to  as  "half  the  time  of  traditional  acquisition".  It  will  be 
important  to  further  define  "expedited"  in  relation  to  more  traditional  or  deliberate  acquisitions.  More 
importantly,  the  characteristics  of  the  fielded  capability  must  be  incorporated  in  the  definition,  such  as  amount  of 
operational  capability/performance,  warfighter  satisfaction,  safety  of  operations,  manufacturability,  and 
sustainability  (repairs,  parts,  etc).  This  report  assumes  "rapid"  or  "expedited"  is  less  than  2  years. 


1.2  Problem  Statement 

The  Defense  Science  Board  (DSB)  Task  Force  on  the  Fulfillment  of  Urgent  Operational  Needs  (July  2009)  identified 
more  than  20  rapid-reaction  programs  and  organizations  addressing  DoD  urgent  warfighter  needs.  A  subsequent 
DSB  study  found  that  urgent  needs  programs  spent  more  than  $50  billion  between  2005-2009,  and  that  urgent 
needs  should  be  considered  a  critical,  ongoing  DoD  institutional  capability.  In  other  words,  "urgent"  is  becoming 
the  new  "normal."  In  March  2011,  the  GAO  report,  "DoD's  Urgent  Needs  Processes  Need  a  More  Comprehensive 
Approach  and  Evaluation  for  Potential  Consolidation,"  found  that  DoD  has  taken  steps  to  improve  fulfillment  of 
urgent  needs,  but  requires  a  common  approach  for  addressing  urgent  needs.  This  recent  2011  GAO  report 
documents  at  least  31  entities  that  manage  urgent  needs  and  expedite  the  solutions  to  address  them.  Thus,  the 
problem  is  a  lack  of  a  framework  that  captures  best  practices  for  expedited  SE  across  these  rapid  acquisition 
organizations,  and  this  research  proposes  a  framework  for  potential  application  across  traditional  acquisition. 


1.3  Description  of  the  Research,  SERC  RT-34 

This  SERC  project  on  Expedited  Systems  Engineering  examined  the  systems  engineering  and  engineering 
management  practices  as  applied  to  rapid  capability  and  urgent  needs  as  developed  in  response  to  changing 
threats.  The  successful  techniques  seen  in  rapid  development  and  prototyping  must  scale  to  larger,  more 
complex,  supportable,  and  sustainable  weapon  systems.  The  systems  engineering  processes  and  practices  applied 
to  urgent  needs  should  provide  for  innovative  conceptual  solutions,  quickly  prune  the  design  space,  and  identify 
appropriate  designs  that  can  deliver  warfighting  capability  expeditiously. 

The  purpose  of  the  research  was  to  explore  and  develop  a  scalable  expedited  SE  framework  for  hybrid  programs, 
i.e.,  those  exploiting  rapid  development,  but  with  the  intent  to  have  traditional  lifecycle  considerations  for 
deployment,  maintainability,  reliability,  adaptability  and  sustainment.  Likewise,  this  new  SE  framework  would  be 
applicable  to  more  traditional  acquisition  programs  with  a  desire  to  incorporate  scaled  rapid  development  best 
practices.  The  focus  is  on  systems  engineering,  not  acquisition  processes.  In  other  words,  the  framework  should 
be  easily  tailorable  or  scalable  to  optimize  for  the  circumstances  of  the  program  in  question. 

RT-34  research  began  September  1,  2011,  with  the  predominance  of  the  data  collection  and  analysis  completed 
by  September  30,  2012,  with  time  designated  until  December  30,  2012,  to  review  the  findings  and  confirm  the 
next  phase  of  research.  The  original  program  plan  consisted  of  the  following  phases: 

•  Phase  1:  Short  planning  phase  to  identify  organizations  practicing  expedited  systems  engineering, 
including  visiting  selected  organizations  and  incorporating  input  from  the  SERC  Research  Council. 

•  Phase  2:  Analyze  current  state  of  the  art  in  expedited  systems  engineering  within  DoD  as  well  as  non-DOD 
and  commercial  sector,  and  developing  a  framework  for  scaling  SE  activities  in  response  to  different 
development  constraints,  such  as  reduced  development  time. 

•  Phase  3:  Prepare  a  plan  for  both  validating  the  framework  on  a  DoD  acquisition  program,  or  plan  for 
follow-on  research. 

•  Phase  4  (Future,  Separate  Funding):  Execute  the  plan  from  Phase  3,  with  research  to  analyze  the 
framework  in  action  and  iterate  it  based  on  observations  and  results  as  applied  to  a  real  program. 
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As  will  be  described  in  this  report,  the  research  evolved  to  consider  the  broader  context  of  the  people,  processes, 
and  products  involved  in  rapid  development.  We  discovered  that  the  framework  was  not  mature  enough  to 
validate  on  a  DoD  acquisition  program.  Additional  research  questions  arose  that,  once  addressed,  could  enable 
the  validation  as  originally  intended. 


1.4  Methodology 

A  recent  research  paper  (Lane,  Boehm,  Bolas,  Madni,  &  Turner,  2010)  on  "Critical  Success  Factors  for  Rapid, 
Innovative  Solutions,"  posed  a  series  of  questions  on  potential  program  success  factors.  That  paper  concluded 
that  successful  innovative  organizations  share  certain  characteristics: 

•  Driven  by  business  value 

•  Take  calculated  risks 

•  Proactive  management  and  small  agile  teams 

•  Look  for  solution  patterns  that  can  be  re-used 

•  Follow  concurrent  engineering  practices 

•  Provide  culture  and  environment  that  supports  innovation. 

Research  on  success  factors  and  factors  categories  is  well  documented  for  varied  purposes  in  the  Industrial 
Engineering  community.  Various  types  of  success  factors  (Van  Scoter  &  Doolen,  2011)  have  been  documented  in 
project  planning,  control,  monitoring,  building,  project,  management,  organization/people,  external  environment, 
process,  and  technical  considerations. 

Based  on  initial  literature  review,  questions  and  input  from  the  SERC  Research  Council  and  personal  experience,  a 
list  of  guiding  questions  was  identified  for  unstructured  interviews  with  subject  matter  experts  (SMEs)  working  at 
headquarter  level  rapid  development  organizations.  After  the  first  week  of  research,  it  was  obvious  that  the  lines 
of  inquiry  could  be  clustered  into  3  categories  of  people,  process,  and  product.  As  a  result,  there  was  some 
refinement  to  these  37  questions  and  re-grouping  into  aspects  of  the  people/  organization,  the  systems 
engineering  and  engineering  management  processes  and  the  solution/  product.  The  re-grouped  questions  were 
used  for  the  remaining  discussions. 

For  example,  some  questions  were: 

•  "How  do  you  select/  design  the  team"?  (people) 

•  "What  is  the  formality  of  engineering  documentation?"  (process) 

•  "How  do  you  translate  prototypes  to  operational  use?"  (product) 

The  team  targeted  those  organizations  who  have  been  acquiring  Joint  Urgent  Operational  Need  Statement 
(JUONS)  solutions  or  who  had  expertise  in  aspects  of  rapid  non-traditional  acquisition.  Predominantly,  the 
organizations  were  either  Government  defense  acquisition  offices  or  select  defense  industries.  The  organizations 
were  identified  by  those  known  to  the  SERC  sponsors,  the  SERC  Research  Council,  and  the  researchers;  those 
identified  in  the  DSB  and  GAO  reports;  and  via  referrals  from  other  rapid  organizations. 

Initially  our  list  included  a  variety  of  organizations  from  government  and  industry,  as  the  original  research  plan 
called  for  us  to  analyze  the  start  of  the  art  in  expedited  SE  within  the  DoD  and  commercial  sector.  As  time  went 
on,  we  narrowed  the  focus  to  government  rapid  acquisition  offices,  as  these  more  closely  represented  the  user 
community  of  interest  to  the  sponsor  and  were  more  consistent  with  the  future  research  intent  to  validate  an 
expedited  SE  framework  on  a  DoD  acquisition  program. 
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Some  of  these  included: 

•  Air  Force  Rapid  Capabilities  Office  (RCO) 

•  U.S.  Army  Prototype  Integration  Facility  (PIF) 

•  An  aerospace  industry  innovation  lab 

•  A  small  satellite  development  company 

•  NASA  Goddard  Space  Flight  Center 

•  Big  Safari  (Program  office  and  USAF  Program  Executive  Office,  ISR-SOF) 

•  Air  Force  Research  Lab  (AFRL)  Center  for  Rapid  Product  Development 

•  SOCOM  Research  and  Development  Acquisition  Center  (SORDAC). 

The  interview  notes  were  coded  and  tagged  based  on  patterns  of  consistent  responses.  Using  the  interview 
notes,  an  iterative  qualitative  analysis  was  performed  using  ATLAS.ti™  software  in  an  effort  to  further  explore  the 
data  for  hidden  connections  and  emergent  trends.  Transcribed  notes  from  organizations  were  assigned  to  an 
ATLAS.ti™  hermeneutic  unit.  Using  the  observations  derived  through  the  interview  process  as  codes,  each 
document  was  coded  appropriately  against  key  words  and  phrases.  The  codes  were  organized  into  families  of 
Product,  Process,  and  People,  which  matched  the  clusters  observed  from  the  discussions.  Originally  20 
observations  were  first  identified,  aggregated  down  to  11.  This  "open  coding"  of  labels  is  an  important  part  of  the 
analysis  concerned  with  identifying,  naming  or  labelling,  categorizing,  and  describing  phenomena  found  in  the 
discussion  notes.  In  this  case,  the  theory  is  a  set  of  principles  for  successful  DoD  rapid  development,  with  a  focus 
on  systems  engineering. 

This  technique  is  used  in  many  inductive  qualitative  approaches,  such  as  grounded  theory.  Grounded  theory  is  a 
type  of  qualitative  research  methodology  that  allows  theories  to  emerge  from  collected  data.  This  collection  of 
data  comes  from  the  notes  during  discussions  with  the  leadership  of  the  "rapid"  organizations— essentially 
experts  in  the  field— to  discern  what  made  them  successful  and  discover  what  drove  their  processes.  The 
research  follows  a  systematic,  yet  flexible  process  to  collect  data,  code  and  analyze  the  data,  make  connections, 
and  see  what  theories  can  be  generated. 

Based  on  observation,  interviews,  and  literature,  a  series  of  observations,  or  principles,  begins  to  emerge  that 
reflects  a  framework  of  rapid  development.  While  we  originally  proposed  a  stacked  model  with  increasing  rigor, 
we  finally  elected  to  not  infer  any  precedence,  hierarchy,  or  increasing  systems  engineering  rigor.  For 
presentation  in  this  report,  we  break  our  findings  into  three  groups: 

•  Group  1:  Direct  Responses, 

•  Group  2:  Direct  Observations,  and 

•  Group  3:  Inferred  Organizational  Characteristics. 

The  Direct  Responses  in  Group  1  provide  an  efficient  affinity  across  all  answers  to  our  37  questions.  This  set 
represents  the  top  11  principles  that  emerged  from  the  ATLAS.ti  analysis,  and  are  called  "Organizational  Best 
Practices".  We  observed  strong  evidence  across  the  site  visits  for  all  11  observations.  In  the  process  of  validating 
the  responses  with  the  SMEs,  reviewing  against  literature,  and  socializing  the  observations  with  others  in  the 
rapid  and  traditional  community,  we  realized  that  the  11  observations  are  "not  new"  and  do  not  appear  to  be 
unique  to  rapid  programs.  We  characterize  these  as  practices  that  can  be  found  in  both  rapid  and  traditional 
development  programs. 

Group  2  "Flexible  Acquisition  Practices"  reflects  Direct  Observations  that  emerged  from  the  site  visits,  but  were 
not  direct  responses  to  the  prepared  questions.  This  group  collects  observations  based  on  flexibility  in  acquisition 
practices,  whether  it  be  contract  type,  finance  type,  program  management  approach,  documentation  required, 
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level  of  insight/oversight,  etc.  These  practices  are  seen  as  critical  for  a  rapid  business  environment.  We  originally 
did  not  seek  out  different  types  of  rapid  categories,  or  strictly  acquisition  management  (such  as  contracting) 
findings;  in  fact,  a  ground  rule  of  the  research  was  that  we  would  not  seek  to  change  the  existing  DoD5000 
process.  However,  these  observations  about  the  acquisition  environment  emerged  from  the  site  visits  and  could 
not  be  ignored. 

Group  3,  the  set  of  "Inferred  Organizational  Characteristics,"  represents  aspects  of  a  "Go  Fast"  Culture  seen  in  the 
rapid  organizations.  This  is  where  rapid  organizations  start  to  differ  from  traditional  ones,  and  there  is  a  shift  in 
energy,  commitment,  and  knowledge.  These  findings  are  motivated  by  an  analysis  of  effective  enterprise  culture 
and  business  agility  literature. 

The  research  methodology,  sites  visited,  and  list  of  questions  used  for  the  site  visits  are  provided  in  Appendix  A. 


1.5  Findings  and  Analysis 

The  Direct  Responses,  Direct  Observations,  and  Inferred  Characteristics,  along  with  a  parsimonious  set  of 
Recommendations  are  summarized  in  Table  1-1.  Some  could  claim  these  ideas  are  not  new,  and  that  these 
principles  have  been  seen  or  demonstrated  in  some  classified  offices  or  a  few  select  acquisition  organizations 
throughout  DoD. 

There  is  an  excitement  and  "vibe"  to  these  organizations  that  cannot  be  captured  from  Table  1-1.  These 
organizations  appear  to  love  what  they  are  doing,  probably  because  they  feel  closer  to  the  need,  the  warfighter, 
and  their  ability  to  successfully  deliver  technological  solutions.  Likewise,  this  excitement  is  contagious 
throughout  the  organizations  and  fosters  leadership,  a  rapid  culture,  decision  making  skills,  and  lifecycle 
expertise.  Data  analysis  of  the  site  visits  (resulting  in  the  Direct  Responses)  can  be  found  in  Appendix  B. 


1.6  Expedited  SE  Framework 

Based  on  the  site  interviews,  the  coding/tagging  of  the  interviews  notes,  extensive  literature  review,  and  the 
knowledge  of  the  research  team,  the  Expedited  SE  Framework  began  to  develop.  Three  groups  emerged. 

•  Group  1  -  Direct  Responses  has  direct  correlation  to  interview  responses  of  rapid  organizations,  and 
confirms  best  practices  in  project  management  and  lean  development  systems. 

•  Group  2  -  Direct  Observations  captures  practices  of  rapid  organizations  that  "live  in  a  rapid  world,"  that 
take  an  integrated  approach  to  rapid  development,  by  leveraging  people,  process  and  product 
appropriately,  as  well  as  seek  out  and  monopolize  flexible  acquisition  practices. 

•  Group  3  -  Inferred  Organizational  Characteristics  signifies  a  shift  in  energy,  commitment  and  knowledge  in 
rapid  organizations,  demonstrating  the  culture  of  effective  organizations  and  those  seen  in  the  agility 
literature. 

See  Figure  1-1. 
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Table  1-1:  Summary  of  RT-34  Observations,  Findings,  and  Recommendations 


Direct  Responses 


Recommendations 


1.  Use  Mature  Technology  -  Focus  on  the  State  of  the 
Possible 

2.  Incremental  Deployment  (Development)  is  Part  of  the 
Product  Plan 

3.  Strive  for  a  Defined  Set  of  Stable  Requirements  Focused 
on  Warfighter  Needs 

4.  Work  to  Exploit  Maximum  Flexibility  Allowed 

5.  Designing  out  All  Risk  Takes  Forever.. .Accept  Some  Risk 

6.  Keep  an  Eye  on  "Normalization" 

7.  Build  and  Maintain  Trust 

8.  Populate  Your  Team  with  Specific  Skills  and  Experience 

9.  Maintain  High  Levels  of  Motivation  and  Expectations 

10.  The  Government  Team  Leads  the  Way 

11.  Right-size  the  Program  -  Eliminate  or  Reduce  Major 
Program  Oversight 

Summary:  Rapid  requires  an  integrated  approach:  People 
making  judgments,  Processes  for  task  reductions,  and  Product 
aspects  focused  on  rapid  objectives. 


Ensure  that  projects  utilize  prioritized 
organizational  and  project  best  practices 

Train  program  managers,  engineers  and 
contracting  officers  in  organizational  and  cultural 
best  practices,  real-time  management  approaches, 
and  the  different  flexibilities  that  exist. 

Encourage  and  enable  programs  to  intensively 
share  knowledge,  have  a  risk-focused  culture,  and 
create  an  ambidextrous  organizational  structure 

Share  knowledge,  experience,  mechanisms  and 
lessons  learned  across  programs  and  organizations 

Quantitatively  monitor  progress  in  expediting  SE 
processes,  and  use  the  measurements  to  improve 
both  schedule  acceleration  and  the  ability  to 
estimate  it 

Use  DOD  rapid  organizations  as  a  testbed  to 
introduce  digitally-enabled  solutions 

Develop  the  acquisition  workforce  using  rapid 
programs  to  provide  full  lifecycle  insight  and 
hands-on  experience. 


Intense  and  efficient  knowledge-sharing  is  used  to 
enable  stabilization  and  synchronization  of 
information 

Rapid  organizations  are  characterized  by  a  risk- 
tolerant  culture 

Rapid  organizations  are  structured  for  ambidexterity 
with  a  balance  between  exploration  and  exploitation. 
Rapid  Development  organizations  exercise  Real-time 
Management. 


Direct  Observations 

Not  a  single  Rapid,  but  many  different  flexible  Rapids  with 
flexible  lanes  of  acquisition  and  business  practices. 

Inferred  Characteristics 


1.7  Report  Organization 

The  next  three  chapters  (Chapter  2-4)  detail  each  of  three  Groups  from  the  Framework  (Figure  1-1).  Chapter  5 
offers  conclusion  and  some  future  research  and  application  considerations.  An  epilogue  to  the  research,  based  on 
a  workshop  at  the  16th  Annual  Practical  Software  and  Systems  Measurement  (PSM)  User's  Group  Conference,  is 
provided  in  Chapter  6. 

Appendix  A  describes  the  methodology  with  the  site  visits  conducted  and  questions  asked  of  the  SMEs.  Appendix 
B  captures  the  ATLAS. ti™data  analysis.  Relations  to  other  SERC  RTs  is  provided  in  Appendix  C  and  details  on  some 
integrated  tools  with  application  to  expedited  SE  can  be  found  in  Appendix  D.  Appendix  E  is  a  paper  that 
describes  an  application  of  schedule-driven  project  estimation  (using  CORADMO). 


Contract  Number:  H98230-08-D-0171 


Page  19 

Report  No.  SERC  - 2012- TR- 034 


TO  0015,  RT034 


UNCLASSIFIED 


Inferred  Characteristics 

"Go  Fast  Cultural  Best  Practices" 


Real-Time  Management 

Intense 

Knowledge  Sharing 

Risk-Tolerant 

Culture 

Organizational 

Ambidexterity 

Direct  Observations 

"Rapid  Best  Practices" 


Flexible  Acquisition 
Practices 

Contracts,  Finance,  Hiring, 
Incentive/Reward  System 


Direct  Responses 

"Organizational  Best  Practices" 


Integrated  Approach  to  Expedited  Work 

PEOPLE 

Trust,  Motivation 
Skills,  Culture 

PROCESS 

Tailor/Scale,  Requirements, 
Risk  Management,  Tech  Debt 

PRODUCT 

Mature  Tech, 
Iterative,  Affordable 

Shift  in  energy, 
commitment,  and 
knowledge. 


Business  practices  and 
leadership  drive  the 
"go-fast"  culture. 


Common  practices, 
for  rapid  or  non-rapid 
development. 


Figure  1-1:  RT-34  Expedited  SE  Framework 
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2  Direct  Responses  for  Organizational  Best  Practices  (Group  1) 


2.1  Introduction 

This  chapter  introduces  the  core  findings  from  the  site  visits  and  interviews,  and  places  those  in  the  context  of 
expedited  acquisition  and  work  reduction  techniques.  Some  may  read  this  chapter  and  see  the  core  findings  as 
"not  new."  In  the  current  program  management  and  lean  product  development  communities,  indeed,  many  of 
these  findings  have  been  observed  in  the  literature  (Morgan  &  Liker,  2006;  Oehmen,  2012).  Similarly,  for  those 
who  practice  DoD  acquisition  in  some  niche  areas  (laboratory  or  operational  prototyping,  classified  acquisition  or 
platform  modification  shops),  some  of  these  findings  may  also  be  commonplace.  Many  organizations  -  traditional 
and  rapid,  are  starting  to  use  more  of  these  practices.  This  chapter  focuses  on  Group  1  of  the  framework. 


2.2  Observations  from  Site  Visit  Discussions 

Appendix  A  describes  the  methodology  for  the  selection  of  site  visits  and  the  questions  posed  to  organizations 
and  individuals  conducting  rapid  development  and  practicing  expedited  systems  engineering.  Appendix  B 
describes  the  data  analysis  conducted  on  the  notes  from  these  discussions. 

The  following  practices  are  a  set  of  consistent  observations  that  have  been  coded  from  discussion  notes  from  the 
site  visits.  Over  30  discussions  were  conducted  from  25  site  visits,  including  a  number  of  informal  discussions  and 
panel  presentations  at  conferences. 

This  section  describes  the  "Group  1"  direct  responses  from  the  site  visit  discussions  as  analyzed  and  derived  from 
22  full  site  visit  discussions,  which  excludes  inputs  from  conferences.  Appendices  A  and  B  further  describe  the 
process.  There  are  11  direct  responses. 

In  summary,  one  major  finding  emerged  that  reflects  how  all  of  the  direct  responses  combine  to  create  an 
integrated  approach  to  rapid  development.  We  did  not  see  just  one  process,  or  one  product,  that  resulted  in 
expedited  systems  engineering.  In  fact,  the  responses  were  much  broader  about  the  context  of  the  overall 
environment,  the  warfighter  need,  the  acquisition  practices,  and  the  combination  of  people,  processes  and 
product  coming  together  for  success. 

Summary  Observation:  Rapid  requires  an  integrated  approach:  People  making  judgments.  Processes  for  task 
reductions,  and  Product  aspects  focused  on  rapid  objectives. 


2.2.1  Product  Practices 

Smart  decisions  on  the  system  solution  can  greatly  lower  cost  and  expedite  product  delivery.  But  the  mindset 
must  be  to  reuse  and  repurpose,  to  the  maximal  extent  possible,  mature  technology  and  legacy  systems  with  an 
eye  toward  reduced  complexity. 

Observation  1:  Use  Mature  Technology  -  Focus  on  the  State  of  the  Possible 

•  Focus  on  integration  of  mature  technologies 

•  Reuse  existing  capabilities,  platforms,  etc.  -  especially  if  they  are  flight-certified 

•  Reduce  Complexity 
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In  rapid  acquisition,  untested  and  unproven  technology  poses  an  enormous  risk  to  system  success.  Unlike  most 
traditional  acquisition  programs,  there  is  no  time  for  technology  to  mature,  in  other  words,  no  time  for  schedule 
slips  due  to  immature  technology  struggling  to  develop.  To  avoid  this  pitfall,  most  rapid  programs  focus 
engineering  efforts  on  the  interfaces  required  to  blend  multiple  existing  technologies  into  a  system  capable  of 
providing  the  desired  set  of  capabilities.  Another  aspect  of  rapid  is  modifying  an  existing  platform  or  simply 
adding  subsystems  and  components.  Program  teams  stay  abreast  of  emerging  technology  and  leverage  the  work 
done  by  industry  and  other  military  programs.  They  then  engineer  a  system-of-systems  solution  to  meet 
requirements.  This  bounds  their  design  space  within  the  state  of  the  possible  -  that  has,  in  part,  already  been 
proven.  In  Technology  Readiness  Level  (TRL)  terms,  this  means  nothing  less  than  a  TRL  6,  preferably  7  or  8.  This, 
in  combination  with  the  concepts  of  reuse  and  incremental  development,  allow  these  teams  to  field  quickly,  but 
generationally  provide  more  and  more  capability. 

In  many  cases,  the  urgent  need  requests  are  to  satisfy  a  newly  emerged  operational  requirement.  These 
organizations  perform  an  abbreviated  analysis  of  alternatives  (AoA)  to  determine  how  best  to  meet  that  need.  By 
the  time  the  request  arrives  at  one  of  these  organizations,  current  technical  capabilities  indicate  a  materiel 
solution  can  be  developed  for  the  warfighter.  Often  the  concept  of  delivering  a  partial  solution  is  driven  not  only 
by  time,  but  also  by  technical  maturity.  While  a  laboratory  may  have  proven  that  something  is  feasibly  possible,  its 
integration  into  the  operational  battlefield  typically  is  several  years  away  until  it  matures.  However,  by  working 
closely  with  the  end  user,  the  lab  may  be  able  to  get  new  technology  into  the  hands  of  the  warfighter  sooner  as 
both  a  test  bed  and  a  way  to  meet  operational  needs.  The  lab  can  also  leverage  existing  components  and 
integrate  them  in  a  new  or  innovative  way,  these  organizations  are  able  to  provide  an  equivalent  or  interim 
solution  in  short  order.  In  this  environment,  the  warfighter  is  given  something  to  use  now,  and  as  technology 
matures,  they  can  expect  greater  capability  in  the  future  that  incorporates  the  experience  collected  in  the  field. 

Another  essential  characteristic  of  rapid  product  development  is  the  reuse  of  existing  technical  capabilities.  This 
is  further  improved  when  existing  technical  capabilities  are  integrated  onto  existing  platforms.  A  great  example  is 
a  recent  modification  one  organization  performed  to  improve  a  small  fleet  of  Combat  Search  and  Rescue  (CSAR) 
helicopters.  Rather  than  develop  a  new  forward  looking  infrared  radar  (FUR)  pod  or  lift  hoist,  the  team  examined 
the  operational  requirements  requested  by  the  user  and  identified  existing  technology  currently  being  installed  by 
the  US  Army  on  Department  of  State  and  Federal  Bureau  of  Investigation  helicopters.  This  reuse  approach  saved 
the  program  office  over  half  of  the  potential  contracted  price  (about  $3M  saved)  and  12  months  of  schedule.  In 
addition,  the  customer  also  received  a  Government-owned  technical  data  package  (TDP)  because  the  design  and 
engineering  work  was  done  in-house.  Another  significant  reduction  in  time  for  this  example  was  the  time  saved 
in  flight  test  by  using  equipment  that  had  already  been  flight-certified.  This  approach  created  huge  efficiencies  in 
their  flight  test  program  as  many  test  requirements  had  already  been  accomplished  by  a  previous  program. 

However,  this  observation  should  be  placed  in  context  to  the  lanes  of  acquisition,  discussed  in  Chapter  4. 
Laboratory  prototypes  tend  to  push  technology  as  greatly  improved  ways  to  provide  existing  capability  or  provide 
new  capabilities.  While  we  observed  that  schedule  demands  often  resulted  in  technologically  mature  solutions, 
the  use  of  immature  or  state-of-the-art  technologies  would  be  appropriate  when  no  other  concept  exists,  or  time 
urgency  can  be  relaxed  or  when  the  risk  is  worth  it.  Essentially,  choice  of  mature  technology  is  another  decision 
that  can  be  approached  through  a  balanced  risk  management  process. 

Observation  2:  Incremental  Deployment  (Development)  is  Part  of  the  Product  Plan 

•  "Generational  development"  -  plan  for  technology  maturity,  advancement,  and  cycles 

•  Look  for  unpredicted  outcomes 

•  Prototyping/Test  infrastructure 
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Part  of  the  agreement  of  accepting  a  partial  solution  may  also  include  the  plan  for  incremental  development. 
When  this  concept  is  decided  upon  from  the  beginning  of  any  development  program,  it  enables  "generational 
development"  -  an  intentional  plan  for  technology  maturity,  advancement,  and  cycles  of  growth.  This  may  be 
done  by  using  open  architectures,  and  modular  concepts,  clearly  defining  system  interfaces,  and  utilizing  industry 
standards.  When  planning  for  incremental  growth  in  platform  capabilities  from  the  start,  particular  systems  level 
planning  is  put  into  place.  This  approach  allows  known  or  unknown  technical  improvements  to  be  more  easily 
integrated  into  the  baseline  system— providing  faster  upgrading  and  an  enhanced  ability  to  share  system  level 
data.  Overall,  this  approach  will  extend  the  system  lifecycle  and  enhance  its  ability  to  flexibly  meet  the  needs  of 
an  ever-changing  technical  and  operational  environment. 

As  rapid  organizations  march  through  their  development  programs,  many  are  constantly  looking  for  unpredicted 
design  outcomes.  In  one  organization,  during  the  latter  stages  of  product  development  a  specific  set  of  questions 
was  asked:  Who  else  could  use  this?  How  else  could  it  be  used?  What  does  this  enable  next?  How  could  this  be 
used  against  us?  This  series  of  questions  put  this  team  in  the  right  mindset  to  further  the  development  and 
utilization  of  their  products  and  think  of  the  application  beyond  the  immediate  urgent  need  and  into  a  future 
program  of  record  or  other  user  requirement. 

One  observation,  not  included  in  the  SME  questions,  was  physically  observed  -  the  infrastructure  to  best  support 
iterative  development.  During  our  visits,  many  organizations  had  the  ability  and  facilities  to  accomplish  rapid 
prototype  and  test.  Iteration  implies  incremental  test  and  feedback.  This  could  be  at  contractor  facilities,  but  was 
generally  observed  in  government  owned-government  operated  facilities.  These  rapid  organizations  had  the 
ability  to  test,  or  had  good  working  relations  with  test  organizations  or  had  their  own  test  organizations.  Thus, 
these  organizations  had  the  infrastructure  to  experiment  on  target  hardware,  prototype  solutions,  and 
incrementally  develop,  test,  deliver  and  get  feedback. 


2.2.2  System/Product  Summary 

In  retrospect,  these  two  observations,  the  use  of  incremental  development  and  the  use  of  mature  technology, 
attempt  to  reduce  system  complexity.  While  the  concept  of  system  complexity  was  not  specifically  mentioned 
during  SME  discussions,  we  consider  this  a  semantically  equivalent  finding.  Complexity  can  be  measured  by  the 
type  and  number  of  components  within  a  solution,  the  degree  of  interconnections  of  those  components  and  the 
type  and  amount  of  coupling  between  the  components.  Often  by  reusing  components,  the  focus  becomes  on 
quickly  integrating  these  components  with  standard  connections. 

Depending  on  the  design,  we  saw  the  integration  of  new  components  with  other  new  components  and  new 
components  with  old  (legacy)  components/  platforms.  Having  well  defined  interfaces  allows  new  components  to 
be  more  easily  integrated  with  other  new  components.  For  example,  USB  interfaces  allow  microprocessors  to  be 
integrated  easily  with  peripherals.  In  another  example,  Operationally  Response  Space  (ORS)  developed  a  plug- 
and-plug  space  vehicle  bus  for  future  fast  and  simple  integration  of  different  payloads. 

Many  assume  that  reuse  of  existing  components  will  reduce  risk  and  deployment  time  directly.  Rather,  a  different 
risk  is  realized  -  the  risk  of  using  a  component  in  a  potentially  different  application,  with  an  unknown 
requirements  and  development  (and  manufacturing)  history.  While  urgent  needs  may  require  mature 
technologies  and  non-development  items  (NDI)  or  COTS,  there  may  be  a  new  set  of  issues  for  any  rapid  solution 
that  may  be  enduring.  An  open  issue  that  requires  further  research,  proposed  in  Chapter  5,  is  the  balance 
between  long  term  sustainability,  affordability,  and  flexibility  for  future  iterations. 

Standard  interfaces,  both  internal  and  external,  still  present  challenges  to  rapid  solutions  and  interoperability. 
Interoperability  has  been  highlighted  as  a  problem  within  the  Department  of  Defense  (DoD)  since  1965  (Ford  et  al, 
2007).  One  of  the  most  used  definitions  for  interoperability,  from  an  outdated  Joint  Publication,  is  the  ability  of 
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systems,  units,  or  forces  to  provide  services  to  and  accept  services  from  other  systems,  units,  or  forces  and  to  use 
the  services  so  exchanged  to  enable  them  to  operate  effectively  together"  (Ford,  2008).  Some  standards  facilitate 
interoperability;  others  are  challenged  by  consistent  interpretation.  One  related  concept  is  that  of  a 
"Convergence  Protocol"  (USAF  Scientific  Advisory  Board,  2005).  These  are  those  standards  that  are  catalysts  for 
widespread  connectivity  and  capability  enhancements;  one  such  convergence  protocol  is  the  worldwide  impact  of 
TCP/IP. 

Bottom  line,  the  approach  was  to  keep  the  system  solution  as  simple  as  possible,  trading  out  items  that  are  not 
critical  to  success,  and  making  maximum  use  of  mature  technologies  within  an  iterative  development  process. 


2.2.3  Process  Practices 

By  first  focusing  on  validating  requirements,  rapid  organizations  then  exploited  and  executed  their  programs  with 
the  greatest  flexibility  allowed. 

Observation  3:  Strive  Toward  a  Defined  Set  of  Tolerable  Stable  Requirements  Focused  on  Warfighter  Needs 

•  Get  the  requirements  right--everything  you  do  stems  from  them 

•  Capability  based  requirements  rooted  in  customer  derived  "CONOPS" 

•  Use  solid  systems  engineering  (SE) 

•  Expedite  trade  studies  -  then  make  a  decision  and  press  forward 

•  Focus  on  providing  the  "23-80%"  solution 

Defining  stable  requirements  focused  on  the  customer  needs  was  one  of  the  most  frequently  occurring  principles 
during  the  SME  discussions.  Not  only  is  there  not  enough  time  to  do  everything  a  customer  is  asking  for,  but 
customers  often  ask  for  more  than  they  really  need  to  meet  their  operational  objectives.  It  quickly  became 
evident  that  every  one  of  these  organizations  spends  a  significant  amount  of  time  up-front,  face-to-face  with  their 
customer  discussing  requirements  and  operational  context.  They  may  actually  spend  more  time  hashing  out  a 
solid  set  of  requirements  than  they  do  in  actual  design  and  production. 

Our  discussions  brought  to  light  several  frustrations  of  the  requirements  development  process.  Customer 
disconnects  or  unrealistic  expectations  may  emerge  because  customers  are  unaware  of  the  state-of-the-possible. 
Customers  may  not  understand  how  difficult  it  might  be  to  accomplish  their  requests.  Occasionally,  a  customer 
may  have  observed  a  system  used  by  another  entity  and  think  "I  need  one  of  those"— seeing  a  product  versus 
identifying  a  specific  capability.  In  response,  the  rapid  organizations  are  deeply  rooted  in  a  capability  based 
approach  to  requirements  analysis.  This  drives  concepts  of  operations  (CONOPS)  based  analysis,  where  the 
customer  must  clearly  define  specific  needs,  uses,  or  capabilities  for  the  system— in  an  operational  context. 

Equally  important  is  an  effort  to  keep  the  requirements  stable.  Regardless  of  the  scope  of  a  project,  requirements 
creep  will  negatively  impact  the  timeline  of  a  project,  delaying  the  delivery  of  operational  capabilities  to  the 
warfighter.  Further,  requirement  changes  potentially  weaken  the  scope  of  the  project  or  may  negate  any 
perceived  increase  in  baseline  capability.  As  a  tenant  for  rapid  SE,  stabilizing  requirements  starts  with  ensuring  the 
requirements  are  right— in  other  words,  directly  interacting  with  the  customer  to  determine  what  the  paramount 
needs  are  rather  then  satisfying  all-inclusive  wants.  Organizations  that  consistently  execute  rapid  SE  and 
acquisitions  are  rooted  in  high-quality  requirements.  In  essence,  rapid  development  requires  stable 
requirements. 

Rapid  organizations  validate  requirements  early  and  often  with  the  customer  to  determine  needs  based  on 
capabilities  with  the  ultimate  goal  to  deliver  an  effect.  The  acquiring  organization  must  be  willing  to  push  back 
against  unfeasible  requirements,  or  schedule  impacting  requirements,  in  the  interest  of  time.  As  one  senior 
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officer  explained,  "[The  organization  must]  fight  hard  to  have  the  warfighter  make  trades"  to  establish 
requirements  that  are  possible  in  the  desired  timeframe.  Simply  put,  focus  on  valid  requirements  that  can  be  met 
by  the  state  of  the  possible  in  a  short  amount  of  time. 

The  application  of  solid  SE  principles  during  early  requirements  definition  promotes  unambiguous  and  achievable 
requirements.  SE  ideologies  emphasize  relating  requirements  to  specific  design  criteria  and  ensuring  the 
traceability  of  those  requirements  from  the  system-level  downward.  Through  an  interactive  process  with  the 
customer,  the  focus  evolves  to  one  of  concept  refinement,  defining  the  system  trade-space,  developing  system- 
level  and  derived  requirements,  making  appropriate  system-level  tradeoff  decisions  and  critically  searching  for 
problems  and  disconnects.  Applying  these  concepts  at  the  beginning  of  the  project  (and  iteratively  throughout) 
can  foster  achievable  objectives  with  reduced  late-in-phase  design  effort. 

Iterative  development,  as  found  in  Observation  2.2,  strongly  relates  to  this  observation  as  well.  Some  engineers 
will  challenge  this,  saying  that  all  requirements,  especially  for  large  acquisitions,  can  be  defined  and  stabilized. 

This  may  be  true.  Complexity  of  the  technologies,  system  interdependencies,  along  with  tactics,  techniques  and 
procedures  of  the  people  using  the  systems,  and  a  dynamic  threat  environment  may  limit  the  ability  to  stabilize 
requirements  early.  Often  the  discovery  of  what  are  the  real  "100%"  of  the  requirements  will  change,  especially 
if  a  smaller  set  of  requirements  is  first  deployed.  For  the  most  complex  systems,  customers  often  don't  know 
precisely  in  advance  what  they  actually  need,  but  instead  must  iterate  through  a  series  of  experiments  to  sort  out 
both  operational  utility  and  effectiveness  of  a  final  solution. 

The  short  duration  of  rapid  development  projects  naturally  lends  to  more  stability  in  requirements.  Grand 
changes  in  technical  maturity  or  capability  are  not  often  experienced  in  the  short  lifecycle  of  the  project.  There 
are  also  fewer  changes  in  political  administration  (funding),  leadership  (rotating  Colonels)  and  program  personnel; 
each  personality  brings  to  the  project  a  new  perspective  or  priority  than  their  predecessor.  And,  requirements 
stemming  directly  from  urgent  warfighter  needs  are  less  likely  to  change  over  a  short  period  of  time. 

Ironically,  requirements  creep  can  become  a  pitfall  of  regular  customer  involvement  in  requirements  refinement. 
Several  organizations  emphasized  the  necessity  to  fight  requirements  creep  once  stable  requirements  have  been 
established.  However,  stopping  creep  cannot  be  done  at  the  expense  of  customer  and  user  involvement.  In  this 
manner,  an  art  must  be  developed  to  keep  the  user  in  the  loop  without  allowing  for  spurious  changes  to  the 
project  once  underway.  A  chief  task  for  the  Systems  Engineer  should  be  persistent  analysis  of  derived 
requirements  in  conjunction  with  making  difficult  decisions  regarding  system  requirement  trades  and  concept 
refinement.  Part  of  this  process  is  understanding  the  real  need  behind  the  warfighter's  request. 

Rapid  programs  rarely  provide  the  customer  with  100%  of  what  they  ask  for.  SMEs  expressed  the  typical  "80% 
solution"  concept,  but  also  a  more  realistic  (albeit  academic  in  number)  "23%  solution"  in  practice.  By  framing 
the  question  to  the  customer  as  "Instead  of  XX  in  10  years,  I  can  give  you  XY  in  a  few,"  the  user  may  be  more 
inclined  to  agree.  One  expert  with  a  space-focused  organization  commented,  "50%  or  23%  done  quickly  can  be 
very  acceptable."  Often  eliminating  or  modifying  certain  requirements  will  provide  the  warfighter  with  a  viable 
solution  to  a  problem  within  an  expedited,  achievable  timeline  rather  than  a  never-ending  pursuit  of  the  100% 
solution. 

The  requirements  process  often  boils  down  to  the  program  team  getting  the  customer  to  clearly  articulate  exactly 
what  capabilities  they  need  in  the  field.  Just  as  critically,  these  conversations  and  research  by  the  development 
team  help  the  customer  understand  the  performance  and  schedule  risk  of  pushing  the  state-of-the  art  too  far. 
Clearly  understanding  the  capabilities  of  industry  and  current  technology  significantly  improves  customer 
expectations  of  what  can  be  accomplished  in  a  short  amount  of  time.  This  then  shapes  the  product  design  space. 
Then,  after  some  quick  analysis,  the  program  team  can  say,  "A  100%  solution  will  take  4  years.  However,  I  can  get 
40%  of  what  you  want  in  about  9  months  and  it  will  do  X,  Y  and  Z.  Will  that  work?"  The  answer  is  often  an 
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enthusiastic  "yes".  In  this  environment,  an  organization  can  be  seen  as  heroic  for  being  able  to  provide  a  solution 
that  does  two  or  three  things  really  well,  delivered  in  a  short  amount  of  time;  rather  than  providing  the  warfighter 
a  system  that  can  do  those  same  three  things  and  assumedly  several  others  after  five  years  (or  more). 

The  researchers  also  met  with  two  well-known  and  published  examples  of  organizations  that  practice  the 
importance  of  up  front  concept  design  and  requirements:  the  Aerospace  Corporation  Concept  Design  Center 
(CDC)  and  the  Georgia  Tech  Research  Institute  (GTRI)  Collaborative  Visualization  Environment  ("COVE").  Both 
facilities  create  and  evaluate  conceptual  designs  in  a  synergistic,  concurrent,  collaborative  manner  with  a 
collocated  team  of  design  engineers  and  customers  applied  to  a  specific  domain.  The  environment  allows  rapid 
and  simultaneous  requirements  development  and  design  engineering  with  real-time  tradeoffs  and  "what  if" 
analyses. 

This  requires  quite  a  bit  of  pre-work  with 
the  users  to  first  describe  the  purpose  of  the 
exercise  (rapid  agreement  on  requirements) 
and  second  to  narrow  down  the  questions 
to  be  addressed  in  the  exercise.  As  an 
example,  the  CDC  specializes  in  spacecraft 
design  and  has  brought  Air  Force  program  offices  into  the  CDC  to  narrow  down  spacecraft  requirements  for 
development  of  a  Request  for  Proposal  (RFP).  As  more  and  more  users  become  familiar  with  the  CDC,  the  Air 
Force  customer  has  used  the  facility  in  increasing  ways  both  for  concept  development  as  well  as  for  trade  studies 
in  later  lifecycle  phases.  The  GTRI  facility  is  very  similar  for  other  domains.  Both  are  capable  of  classified  projects. 


Decision  Making  Tools  at  the  GTRI 
Collaborative  Visualization  Environment 
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Figure  2-2:  Aerospace  Corporation  Concept  Design  Center  (CDC)  [LEFT] 

&  Georgia  Tech  Research  Institute  (GTRI)  Collaborative  Visualization  Environment  ("COVE")  [RIGHT] 


Observation  4:  Work  to  Exploit  Maximum  Flexibility  Allowed 


•  Tailor  the  acquisition  and  system  engineering  process  to  the  product 

•  Establish  a  clear  and  short  approval  chain 

•  Document  what  is  important  and  decisions  made  -  not  much  else 

•  Use  various  contracting  vehicles  to  accomplish  different  tasks 

It  may  appear  to  the  casual  viewer  that  rapid  organizations  are  the  "Wild  West"  of  the  DoD  acquisition 
community.  However,  solid  acquisition  and  engineering  approaches  to  solving  complex  technical  problems  and 
fulfilling  real  operational  needs  were  consistently  observed.  Because  of  the  specialized  nature  of  each  office, 
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many  have  developed  in-house  processes  adaptable  to  each  new  program.  This  ensures  each  program  office  has 
a  specific  roadmap  leading  it  to  success,  and  each  project  lives  within  its  own  specific  process  and  lifespan. 

In  the  interest  of  time,  these  organizations  ensure  every  acquisition  or  SE  process  they  implement  is  tailored  to 
the  end  product  or  the  desired  effect  to  be  delivered.  Anything  not  required,  deemed  unnecessary,  or  found  to 
be  non-value  added  is  set  aside.  Adhering  to  the  intent  of  DoDI  5000.02,  they  apply  it  to  their  specific  product 
without  excess.  To  the  untrained  eye  it  may  appear  these  organizations  are  skipping  steps  in  the  acquisition 
process.  The  research  indicates  these  steps  are  not  skipped,  but  rather  tailored  to  meet  the  stringent  timelines 
required  to  deliver  product  to  the  warfighter.  For  example,  a  Systems  Engineering  Plan  (SEP)  may  consist  of  only 
two  pages  within  a  higher  level  document,  instead  of  a  30  page  stand-alone  file.  They  use  formal  and  informal 
review  processes,  specifically  performance-based  and  milestone-type  reviews,  with  just  the  right  people  in 
attendance  to  make  go/no-go  decisions  on  the  spot.  The  focus  is  to  document  important  technical  and 
programmatic  information  and  critical  decisions.  There  are  no  documents  produced  for  documentation  sake. 

In  discussion  with  some  organizations,  it  became  evident  their  approval  chain  for  reviews  and  program  milestone 
approval  had  been  shortened.  Additionally,  the  approval  chains  were  clearly  defined.  In  most  of  these 
organizations,  there  are  very  few  extraneous  persons  in  the  review  chain  that  do  not  have  some  sort  of  approval 
authority  or  intrinsic  value  added  (such  as  legal  or  contracting). 

The  brevity  of  these  approval  chains  often  stems  from  a  Program  Management  Directive  (PMD)  outlining  the 
decision  making  authority  within  these  organizations.  This  document  can  outline  specific  positions  with  approval 
authority,  typically  pushing  it  down  to  a  lower  level  of  responsibility  within  the  organization,  shortening  the 
approval  chain  and  reducing  the  time  required  to  make  programmatic  decisions.  Some  of  this  brevity  may  also 
stem  from  the  classification  level  of  the  project,  literally  preventing  some  personnel  from  participating.  Finally, 
program  size  may  keep  budgets  under  Major  Defense  Acquisition  Program  (MDAP)  thresholds. 

An  often  frustrating  part  of  the  acquisition  process  is  bureaucracy.  SMEs  indicated  that  occasionally  some 
individuals  believe  they  need  to  be  part  of  a  review  process  ostensibly  because  of  their  position,  leadership  or  not, 
which  can  become  a  road  block  for  the  program  team.  Conversations  reveal  this  behavior  may  be  caused  by 
personal  agendas,  need  for  a  feeling  of  empowerment  or  importance,  or  simply  because  the  process  has  always 
been  done  a  certain  way.  In  rapid  programs,  if  someone  does  not  have  value  to  add,  they  are  not  included.  This 
was  not  to  the  exclusion  of  participation,  but  value  added  by  personnel  directly  or  indirectly  involved  in  the 
process  is  critically  analyzed.  This  analysis  helps  avoid  the  pitfall  of  people  merely  adding  time  to  the  process  and 
pushing  back  on  the  review  process  without  bona  fide  authority  to  do  so. 

Another  practice  is  to  combine,  not  skip,  program  level  reviews.  For  example,  test  plans,  Technical  Readiness 
Review  Boards  (TRRB),  Safety  Review  Boards  -  if  deemed  low  risk,  can  be  signed  off  at  the  local  level  in  a  single 
review.  This  is  also  applied  to  pre-milestone  decision  reviews  as  well.  This  concept  does  not  indicate  system 
engineering  processes  and  thoroughness  are  brushed  aside.  The  approach  is  to  shorten  the  approval  and  review 
process  timeline  by  combining  review  processes  and  reducing  the  lull  created  by  waiting  for  a  review  process  to 
take  place  -  not  to  diminish  the  quality  of  the  product  or  eliminate  SE  analysis  processes. 

Another  common  trend  is  the  use  of  various  innovative  contracting  vehicles  to  accomplish  different  tasks,  some  of 
which  are  in  place  for  several  years,  for  use  when  needed  or  to  provide  frequently  used  specific  services.  For 
example,  Indefinite  Delivery  Indefinite  Quantity  (IDIQ)  contracts  were  important  to  several  organizations  to 
provide  as-needed  support  on  a  reoccurring  basis.  This  approach  requires  a  special  type  of  contracting  capability, 
referred  to  by  one  organization  as  "creative  contracting".  This  can  only  be  done  by  contracting  officers  who  are 
knowledgeable  about  and  willing  to  investigate  the  art  of  contracting  -  discover  how  something  could  be  put  on 
contract  in  a  way  most  advantageous  to  the  product  and  program  situation— all  within  the  bounds  and  utilizing 
the  full  flexibility  of  the  existing  Federal  Acquisition  Regulation  (FAR)  and  policy. 


Contract  Number:  H98230-08-D-0171 


Page  27 

Report  No.  SERC  - 2012- TR- 034 


TO  0015,  RT034 


UNCLASSIFIED 


Observation  5:  Designing  out  All  Risk  Takes  Forever  -  Accept  Some  Risk 

•  Creative  (and  implementable)  solutions  are  allowed 

•  Mitigate  risk  through  the  use  of  mature  and  proven  technology 

•  Potential  for  failure  is  accepted,  because  providing  something  may  be  better  than  nothing 

•  Determine  the  level  of  risk  the  customer  is  willing  to  accept 

The  rapid  organizations  we  met  with  operate  under  an  uncommon  risk  paradigm  when  compared  to  many  large 
DoD  acquisition  programs.  In  rapid,  the  potential  for  "failure"  through  providing  only  a  partial  or  short  term 
solution  to  the  field  may  be  acceptable,  as  this  may  be  preferable  to  delivering  nothing  at  all.  Rapid  teams  are 
made  up  of  technical  experts  who  cognitively  assess  the  risks  of  different  technical  solutions  throughout  the 
design  process,  sometimes  with  formal  risk  assessment  processes  in  place.  This  idea  of  risk  mitigation  through  use 
of  mature  and  proven  technology  led  several  programs  to  adopt  the  concept  of  demonstrations  or  prototyping 
versus  modeling  as  a  better  use  of  time  and  resources.  The  bottom  line  often  came  down  to  the  level  of  program 
or  technical  risk  the  customer  is  willing  to  accept  and  the  limited  time  available— emanating  from  detailed 
conversations  with  the  customer.  If  the  warfighter  could  utilize  a  partial  solution  and  is  willing  to  take  some 
technical  risks  with  a  prototype  in  the  field,  delivery  times  are  considerably  shortened  and  feasible  solutions  can 
be  arrived  at  allowing  testing  in  the  field  and  real-world  feedback  for  incremental  improvements. 

According  to  Merriam-Webster's  dictionary,  "create"  is  defined  as:  To  produce  through  imaginative  skill 
(Merriam-Webster,  2012).  While  thinking  creatively  is  not  necessarily  commonplace  in  everyday  acquisition,  in 
rapid  acquisition  it  is  absolutely  acceptable  and  quite  often  critical  to  success.  Creative  and  implementable 
solutions  must  be  sought  in  order  to  do  things  rapidly.  Some  of  this  success  hinges  on  expert  understanding  of 
the  design  space,  potential  technical  solutions,  and  the  ability  to  integrate  existing  technologies.  Rapid  programs 
work  through  a  rigorous  design  process,  working  to  identify,  eliminate  and  accept  risks.  However,  attempting  to 
design-out  all  risk  is  a  time  consuming  and  costly  process,  and  not  realistic  if  attempting  to  get  a  solution  out  to 
the  customer  quickly. 

Observation  6:  Keep  an  Eye  on  "Normalization" 

•  Track  your  technical  debt 

•  Do  configuration  management,  even  if  it  is  in  your  engineer's  head 

•  Buy  or  maintain  data  rights  or  a  build-to  spec 

"Normalization"  is  a  term  heard  at  one  of  the  larger  DoD  rapid  acquisition  offices,  but  the  concept  was  reoccurring. 
It  describes  the  transition  of  a  program  from  a  prototype  or  rapid  project  into  a  major  (mainstream)  acquisition 
program.  Most  of  the  organizations  interviewed  typically  work  in  small-rate  production  runs  (a  few  to  less  than  15). 
Thus,  the  investments  required  for  product  implementation  are  minimal  compared  with  a  large  aircraft  program 
predestined  for  a  full  rate  production  phase  and  years  of  sustainment.  However,  as  many  rapid  projects  have  the 
potential  to  become  normalized  or  productized,  it  is  advantageous  for  these  offices  to  keep  their  eyes  on  this 
possibility  and  be  prepared  for  a  full-scale  transition.  Effectively,  this  term  relates  to  a  program  that  was  originally 
schedule-driven,  but  has  been  asked  to  become  lifecycle-optimized  and  to  transition  (usually  after  the  fact)  to  a 
program  of  record.  This  typically  happens  when  an  urgent  need  is  determined  to  be  an  enduring  requirement. 

Technical  debt,  another  term  heard  at  one  of  the  organizations,  is  a  concept  coined  by  Ward  Cunningham  in  the 
early  1990s  as  a  way  to  describe  the  risks  and  compromises  made  in  rapid  development.  He  first  applied  the 
concept  to  software  development.  "Shipping  first  time  code  is  like  going  into  debt.  A  little  debt  speeds 
development  so  long  as  it  is  paid  back  promptly  with  a  rewrite...  The  danger  occurs  when  the  debt  is  not  repaid. 
Every  minute  spent  on  not-quite-right  code  counts  as  interest  on  that  debt.  Entire  engineering  organizations  can 
be  brought  to  a  stand-still  under  the  debt  load  of  an  unconsolidated  implementation,  object-oriented  or 
otherwise."  (Cunningham,  1992) 
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Cunningham  has  recently  commented  that  this  concept  has  been  misinterpreted  and  confused  with  the  idea  that 
you  can  do  sloppy  or  poor  work  up  front  with  the  intention  of  doing  a  good  job  later.  (Cunningham,  2009)  That  is 
not  the  case  with  the  rapid  organization  whose  primary  purpose  is  to  provide  useful  products  to  the  warfighters  in 
the  field.  Providing  a  poorly  executed  product  to  the  field,  however  rapidly,  would  quickly  render  these 
organizations  useless. 

The  technical  debt  concept  allows  these  organizations  to  move  forward  quickly,  before  they  may  fully  understand 
the  problem,  all  the  while  tracking  what  processes  has  been  potentially  skipped  or  tailored  (both  purposeful 
tailoring  or  ad  hoc  tailoring).  The  importance  of  tracking  these  processes  comes  into  play  as  the  program 
matures,  particularly  if  the  program  is  successful  and  is  transformed  into  a  larger  military  procurement  program 
or  new  program  of  record.  It  should  be  noted  that  tailoring  may,  or  may  not,  incur  technical  debt.  If  these 
processes  or  analyses  are  not  tracked,  it  will  be  difficult  to  know  what  work  might  need  to  be  completed  as  the 
program  moves  into  traditional  maturity.  For  example,  on  a  small-scale  small-shop  project,  configuration 
management  may  have  been  done  in  the  engineer's  head.  However,  if  there  is  a  desire  to  mass  produce  an  item, 
configuration  management  and  a  true  product  baseline  will  be  needed.  Knowing  what  systems-level  work  has  or 
has  not  been  accomplished  is  critical  to  successful  transition  to  a  normalized  and  potentially  mass  produced 
product.  It  is  often  difficult  for  rapid  organizations  to  accomplish  this  transition  because  of  the  time  it  takes  to 
overcome  the  technical  debts,  particularly  in  the  documentation  required  for  a  program  of  record. 

This  concept  also  confirms  a  popular  topic  amongst  all  acquisition  and  engineering  programs— data  rights.  Many 
of  these  organizations  specifically  mentioned  the  benefits  of  purchasing  or  maintaining  some  level  of  government 
owned  data.  The  level  of  data  required  varies  between  programs,  but  the  intent  was  consistent:  Have  enough 
data  to  provide  the  ability  to  modify  when  necessary,  maintain  competition,  and  facilitate  a  transition  toward 
normalization. 


2.2.4  People  Practices 

The  discussions  with  SMEs  uncovered  aspects  of  trust,  strong  team  skills,  empowered  leadership  and  a  unique 
culture  with  high  expectations  of  the  team. 

Observation  7:  Build  and  Maintain  Trust 

•  Develop  solid  relationships  and  work  to  maintain  them 

•  Empowered  leadership 

•  Autonomy  for  Program  Managers/Engineers 

•  Consistent  customer  input  &  buy-in  every  step  of  the  way 

Building  and  maintaining  trust  enables  empowered  teams  working  together,  being  allowed  to  make  decisions, 
leaders  standing  behind  their  decisions,  and  dealing  with  success  or  failures  as  they  are  encountered.  This 
building  process  is  a  struggle  at  times  and  may  even  involve  internal  and  external  conflicts.  These  conflicts  must 
be  enriching  experiences,  opportunities  to  learn,  grow,  cooperate,  and  move  forward.  Agile  development  thrives 
in  a  culture  "where  people  feel  comfortable  and  empowered  by  having  many  degrees  of  freedom".  The  scope  of 
trust  is  an  important  element  for  expeditious  behavior  and  extends  throughout  the  organizations  in  acquisition, 
development  and  deployment,  and  particularly  in  the  interfaces  between  these  three.  As  noted  by  P.  Lencioni 
("Five  Dysfunctions  of  a  Team")  and  others,  T rust  is  the  critical  foundation  of  teamwork  without  which  it  is  not 
possible  to  effectively  team  or  collaborate.  Trust  becomes  ever  more  important  amongst  the  parties  at  higher 
levels  of  decision  making. 
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Rapid  organizations  repeatedly  showed  leadership  at  all  levels  providing  top  cover  to  allow  teams  to  focus  on 
executing  the  mission.  These  same  leaders  must  be  empowered  and  trusted  at  the  lowest  level  possible  to  make 
tactical  and  strategic  decisions.  When  decision  making  authority  is  placed  at  a  low  level  it  shortens  the  process, 
reduces  opportunity  for  stall  time,  and  fosters  close  relationships. 

Most  discussions  conducted  circled  back  to  strong  relationships  with  the  customer.  From  this  perspective,  it  was 
vital  to  have  the  customer  consistently  involved  in  the  decision  making  process  and  to  gather  their  feedback  as 
the  process  moved  forward.  This  was  accomplished  in  many  different  ways:  Short-  and/or  long-term  on  site 
customer  representatives,  customer  input  and  regular  conversations  through  reviews,  or  simply  a  close 
relationship  and  coordination  process.  Regardless  of  how  the  customer  was  included  on  the  team,  it  was  clear 
that  trust  in  the  team's  ability  to  deliver  was  vital  to  project  success. 

T rust  is  built  through  expertise,  show  of  confidence,  and  record  of  performance.  On  the  outside,  it  appears 
relationships  exist  on  an  organizational  level.  However,  our  discussions  showed  building  and  maintaining  trust 
within  a  program  team  required  constant  nurturing.  Further,  trusting  relationships  showed  just  as  important 
between  individuals  within  these  teams  as  building  and  maintaining  trust  with  customers  and  senior  leaders.  It 
was  consistently  demonstrated  that  personal  trust  relationships  at  every  level  built  foundations  for  organization 
reputation  and  credibility.  In  addition,  the  existence  of  a  trust  network  appeared  important  for  developing 
connections  inside  and  outside  the  organizations.  Further,  personal  trust  networks  became  intertwined  - 
enhancing  and  extending  the  capabilities  and  connections  between  individuals  and  organizations.  These 
relationships  were  often  sustained  into  future  jobs.  This  helps  quickly  build  trust  by  leveraging  pre-existing  and 
proven  relationships  to  build  new  ones. 

Individuals  build  trust  with  one  another  through  demonstrated  commitment  and  competence.  A  successful 
acquisition  team  must  have  highly  skilled  acquisition  professionals.  But  it  is  only  through  the  consistent 
application  of  those  skills  that  trust  is  built  with  leadership  and  individual  or  organizational  autonomy  is  granted. 
Thus,  not  only  must  the  desire  to  grant  autonomy  or  empowerment  exist  in  the  leader,  it  must  be  earned  by  those 
at  the  lower  level.  It  is  on  the  back  of  established  trust  relationships  with  senior  leadership  and  the  customer  that 
this  autonomy  allows  small  teams  to  rapidly  move  programs  forward. 

Observation  8:  Populate  Your  Team  with  Specific  Skills  and  Experience 

•  Hand  pick  your  team...or  grow  your  own 

•  Acquire  people  with  the  right  education,  experience,  and  personality 

•  Build  the  right  team  for  each  project 

The  SME  discussions  alluded  to  hand  picking  teams  and  developing  specific  skill  sets  as  a  key  aspect  of  success. 
Data  indicated  over  90%  of  the  interviewed  organizations  handpicked  their  staff.  Organizations  identified 
required  skills  needed  for  each  project  and  took  necessary  actions  to  acquire  that  skill  set.  Several  methods  of 
acquiring  these  skill  sets  were  used:  handpick  new  individuals,  grow/groom  current  personnel  (such  as  through 
mentoring,  shadowing,  or  on-the-job  experiences),  hire  contractor  support,  and  reorganize  teams.  For  rapid 
organizations,  these  vital  individuals,  either  of  their  own  accord  or  external  grooming,  become  experts  with  very 
specific  skill  sets  and  a  broad  set  of  experiences.  These  individuals  can  then  apply  their  skill  sets  to  projects  with 
specific  customers,  technologies  or  operational  contexts. 

Several  senior  leaders  interviewed  brought  focus  to  expertise  by  indicating  that  a  vital  trait  of  aggressive  DoD 
acquisition  involves  acute  proficiency  and  depth  concerning  the  application  of  the  so-called  "normal"  acquisition 
process.  In  order  to  tailor  the  applicable  rules  of  acquisition  and  engineering,  team  members  must  first 
understand  what  the  rules  are  and  which  rules  or  processes  apply  to  the  current  situation.  People  with  deep 
roots  and  experience  in  acquisitions,  contracting,  finance  and  engineering  know  what  the  standard  processes  are. 
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They  have  executed  large  and  small  projects  using  various  methods  and  standards.  Thus,  they  are  keenly  aware 
of  the  implications  from  omitting  or  tailoring  a  step  or  the  challenges  in  executing  parallel  development 
processes.  Their  expert  knowledge  of  the  proper  process  allows  them  to  create  a  process  specifically  designed  to 
meet  the  needs  of  their  program. 

One  particular  rapid  organization  was  not  100%  selectively  manned.  As  leadership  determines  both  the  technical 
strengths  of  the  team,  as  well  as  activities  that  bring  staff  personal  reward,  organizations  can  re-evaluate  their 
internal  structure.  When  asked  how  they  organized  the  team  to  account  for  this,  they  stated,  "We  evaluate  our 
team  by  the  strengths  and  skill  sets  we  are  given.  If  we  have  to  reorganize  a  flight  to  meet  the  skill  set  of  the  team 
we  have,  we  do  it.  Finding  out  what  a  team  member  enjoys  and  is  good  at  and  letting  them  work  in  that  area  all 
but  makes  up  for  lack  of  'handpicking'  every  member."  In  some  cases,  a  person  with  the  right  attitude, 
personality,  or  motivation  can  make  up  for  a  lack  of  technical  skill  or  experience.  In  other  words,  this  organization 
was  able  to  make  up  for  a  specific  lack  of  knowledge  and  skill,  by  strategically  leveraging  the  strengths  of  the 
personnel  they  had— even  if  that  meant  moving  personnel  around  as  projects  progressed. 

Besides  the  desire  to  hand  select  personnel,  most  rapid  organizations  required  a  long  term  commitment, 
particularly  for  military  personnel.  Instead  of  the  typical  two  year  job  rotation,  military  members  are  on  three  to 
four-year  controlled  tours— only  released  for  command,  Professional  Military  Education  opportunities,  or  other 
unique  situations.  Organizations  cited  a  desire  to  keep  good  talent  as  long  as  possible,  and  influence  on-the-job 
experience  as  individuals  grew  in  their  ability  to  execute  organizational  processes. 

Observation  9:  Maintain  High  Levels  of  Motivation  and  Expectations 

•  Mindset:  Motivated,  Collaborative,  competitive,  impatient,  creative,  technical,  independent 

•  Mistakes  are  OK,  but  it  is  not  OK  to  repeat  them 

•  Every  member  connected  to  the  mission  and  vision 

As  the  research  team  met  with  rapid  organizations,  a  certain  enthusiasm  was  noticed  abounding  in  the  leaders 
and  personnel— seeming  to  share  a  state  of  mind  that  was  somehow  traditionally  military  and  entrepreneurial  in 
spirit.  The  mindset  of  these  individuals  expressed  a  competitive  nature  born  from  a  unique  skill  set,  an  aggressive 
and  competitive  environment,  and  a  tangible  connection  to  helping  accomplish  an  operational  mission.  They  are 
motivated. 

Through  discussion,  this  motivation  appears  to  emanate  from  three  primary  sources.  First,  there  is  a  direct 
connection  to  an  operational  community.  Working  closely  with  the  end  users  creates  both  a  connection  to  the 
operational  task  at  hand  and  puts  a  face  on  the  customer.  The  team  is  not  just  rushing  to  develop  an  oxygen 
sensor  for  F-22  pilots;  they  are  developing  it  for  Capt  Josh  "Tread"  Saufley,  so  he  avoids  getting  hypoxic  on  his 
next  flight.  Second,  there  is  a  sense  of  urgency.  JUONS  by  their  nature  are  "urgent"  and  of  critical  importance. 
Providing  capability  to  the  field  may  very  well  be  a  matter  of  survival  and  mission  success  for  US  military 
members.  Finally,  the  rapid  nature  of  these  projects  provides  a  tangible  result  not  typically  experienced  by 
members  of  the  acquisition  and  engineering  community.  Members  of  the  rapid  acquisition  community  have  the 
opportunity  to  see  the  full  project  from  concept  definition,  through  development,  and  launch  it  into  operational 
use.  This  concrete  effect  of  seeing  the  fruits  of  labor  utilized  by  its  intended  customer  can  be  very  powerful  and 
help  maintain  sustained  levels  of  motivation-even  through  long  and  arduous  workdays. 

A  unique  environmental  characteristic  observed  in  several  organizations  was  one  in  which  mistakes  are  OK,  but 
not  OK  to  be  repeated.  This  concept  is  vital  to  fostering  a  creative,  collaborative,  and  yet  competitive 
environment.  One  specific  technique  observed  to  hone  organizational  skills  is  a  "debrief  culture".  Originating 
from  the  operational  world  of  reviewing  a  mission,  focused  debriefs  on  team  performance  can  be  extremely 
powerful.  A  debrief  culture  emphasizes  learning  from  mistakes  and  works  to  identify  root  causes  (individual  or 
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organizational)  to  improve  future  endeavors.  Furthermore,  the  debrief  process  may  be  applied  to  iterations  or 
phases  of  current  projects  in  addition  to  a  final  project  debrief.  The  purpose  of  a  focused  debrief  is  to  determine 
what  went  wrong  and  develop  "lessons  learned"  (much  like  a  detailed  heuristic)  to  prevent  the  same  errors  from 
occurring  in  the  next  project  or  subsequent  iterations  of  the  current  project. 

The  debrief  culture  must  be  established  at  the  top  level  of  the  organization  where  leaders  outline  and  enforce  the 
expectations  and  importance  of  the  debrief  process.  A  successful  debrief  methodology  centers  on  developing 
focus  points  derived  from  the  comparison  of  the  project's  intended  objectives  and  the  actual  outcome,  and  then 
investigating  to  determine  the  root  causes  of  the  focus  points.  To  clearly  maintain  motivation  and  expectations 
inside  an  organization,  each  member  needs  to  expect  that  a  thorough  debrief  will  be  conducted  with  the  overall 
goal  of  identifying  the  underlying  cause  of  any  less-than-perfect  outcomes. 

Observation  10:  The  Government  Team  Leads  the  Way 

•  High  level  of  expectations  for  government  personnel  (military  and  civilian)  to  run  programs 

•  Focus  on  full  use  of  government  personnel  capabilities  -  technical  competence  is  expected 

Rapid  organizations  work  hard  to  find  and  hire  military  and  government  experts.  Government  personnel  are 
expected  to  run  the  programs,  often  times  without  a  prime  contractor  or  support  contractors  as  part  of  the 
organization.  Many  of  the  rapid  programs  interviewed  had  a  small  support  contractor  footprint,  if  at  all- 
compared  to  most  major  acquisition  programs.  This  is  not  to  say  they  did  not  employ  or  rely  on  contractors  to 
provide  leadership  or  technical  support  on  a  large  or  small  scale.  However,  when  programs  did  have  a  support 
contractor  workforce,  the  expectation  was  still  the  same:  The  government  engineer,  program  manager, 
operations  representative,  etc.,  was  expected  to  be  the  resident  expert  on  the  program. 

These  government  teams  are  typically  comprised  of  a  set  of  functional  experts  as  a  development  team.  Core 
capabilities  will  exist  on  these  teams  -  a  program  acquisition  officer,  resource/financial  manager,  system 
engineer,  operational  expert,  safety,  and  test  personnel.  Technical  competence  is  the  standard,  not  the 
exception.  It  is  expected  every  member  of  the  team  is  technically  able  to  run  his  or  her  portion  of  the  program. 
They  maintain  awareness  of  activities  and  issues  on  all  aspects  of  the  development  program,  regardless  of 
government  or  contractor  responsibility.  There  is  little  room  for  redundancy. 

In  conversation  with  a  Chief  Engineer  from  a  large  program  office,  the  following  interaction  was  recounted:  At  a 
weekly  internal  program  review  several  support  contractors  were  briefing  status  with  a  handful  of  Lieutenants 
and  Captains  sitting  silent  around  the  room.  One  month  into  the  job  he  asked  the  support  contractors  to  hold 
their  concerns  and  asked  the  Captains  to  brief.  They  could  not.  His  question  back  to  them  was,  "Why  are  you 
here?" 

Observation  11:  Right-size  the  Program-Eliminate  or  Reduce  Major  Program  Oversight 

•  Smaller  Systems  and  Budgets  Receive  Less  Oversight  and  are  More  Stable 

Budgets  are  often  thought  of  as  a  process  principle,  but  it  depends  on  the  context.  One  benefit  many  of  the  rapid 
program  offices  enjoy  is  a  lack  of  size.  When  you  are  to  move  fast,  smaller  is  often  better.  Not  only  do  large 
organizations  create  challenges  to  effective  management  and  full  utilization  of  all  personnel  resources,  they  tend 
to  have  larger  budgets.  Big  programs  and  big  budgets  can  easily  become  targets  for  increased  oversight,  longer 
approval  chains,  and  funding  cuts.  In  this  sense,  being  big  creates  its  own  problems.  Size  becomes  its  own 
enemy. 
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In  this  research,  the  size  of  the  program  budgets  appeared  directly  related  to  the  products  themselves.  The 
design  and  technologies  selected  to  meet  operational  requirements  directly  impact  the  cost  of  the  program.  Sub¬ 
system  product  selection,  interface  complexity,  sustainment  considerations,  and  technical  maturity  all  drive  cost. 
Keep  in  mind  these  organizations  are  focusing  on  the  23-80%  solution,  are  not  going  into  mass  production,  and 
are  not  necessarily  planning  for  long-term  sustainment.  However,  these  organizations  intentionally  take  steps  to 
reduce  the  overall  size  of  their  budgets.  For  example,  the  willingness  to  accept  some  types  of  risk  buys  down  the 
cost  of  the  design,  development  and  manufacturing  efforts.  Costs  (and  risk)  are  also  reduced  by  using  proven  or 
mature  technology.  Utilizing  simple  or  standard  interfaces  can  help  reduce  complexity— reducing  development 
costs. 

As  with  contracting  officers,  many  rapid  organizations  have  dedicated  finance  personnel  who  work  to  manage  the 
cornucopia  of  funding  sources  for  these  projects.  Imagine  the  variety  of  funding  types  coming  into  an 
organization  that  executes  projects  for  all  military  branches,  several  3-letter  government  agencies,  and  foreign 
military  sales.  These  organizations  rely  on  the  flexibility  of  multiple  funding  types.  Some  customers  come  to  them 
with  a  need,  but  arrive  with  funding  from  various  sources  or  not  the  right  color  of  money.  It  takes  a  special  kind 
of  organization  and  finance  officer  to  understand,  acquire  and  execute  these  funds. 


Contract  Number:  H98230-08-D-0171 


Page  33 

Report  No.  SERC  - 2012- TR- 034 


TO  0015,  RT034 


UNCLASSIFIED 


3  Direct  Observations  for  "Rapid  World"  Best  Practices  (Group  2) 


This  chapter  introduces  those  findings  that  reflect  best  practices  of  those  organizations  that  "live  in  a  rapid 
world."  This  is  where  the  leadership  and  business  practices  drive  the  everyday  culture.  We  identify  highly 
flexible  practices  across  various  "lanes  of  acquisition." 

Direct  Observation:  Not  a  Single  Rapid,  But  Many  Different  Flexible  Rapids 


3.1  Lanes  of  Acquisition 

Our  discussions  with  rapid  organizations  indicated  a  number  of  different  definitions  of  Rapid,  as  well  as  a  wide 
range  of  specific  practices  that  were  implemented  to  achieve  Rapid,  and  various  domains  of  product  categories 
that  aligned  to  what  the  rapid  organization  aimed  to  achieve.  It  was  observed  that  various  organizations 
appeared  to  focus  and  operate  within  lanes  of  acquisition,  which  could  be  analogous  to  lines  of  business  in  the 
commercial  sector.  The  lanes  emerged  as  a  result  of  looking  at  the  "3  P's"  seen  in  the  initial  line  of  questioning. 
Four  lanes  were  observed,  shown  in  Figure  3-1.  They  are  defined  at  the  top  level  by  "product"  type,  and  each  has 
a  corresponding  type  of  "process"  (from  highly  tailored  to  following  every  step  of  the  DOD  5000  series)  and  a  set 
of  "people"  attributes  that  work  best  in  that  particular  name.  Thus,  there  is  no  single  ideal  archetype  of  rapid 
acquisition. 
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Figure  3-1:  Lanes  of  DoD  Rapid  Acquisition,  as  Observed  in  RT-34 

The  lanes  of  DoD  rapid  acquisition  we  observed  are  as  follows,  based  on  analysis  of  the  RT-34  site  visits,  and  were 
refined  against  review  of  existing  known  practices  within  the  government  (both  DoD  and  NASA): 

1.  Laboratory  Demonstration  /  Operational  Prototype.  This  category  is  an  activity  to  rapidly  design,  develop, 
and  test  technologies  (which  can  be  an  individual  technology,  component,  subsystem,  or  integrated  system), 
typically  in  a  laboratory  or  rapid  prototyping  environment.  The  intent  is  to  demonstrate  the  technology  first 
in  the  lab  or  a  test  environment,  with  eventual  demonstration  in  the  field  or  to  at  least  evaluate  the  demo  for 
military  utility.  Evaluation  can  be  in  a  realistic  or  simulated  operational  environment.  Some  designs  may  be 
"one-off"  designs  developed  in  the  laboratory.  A  warfighter  need  may  result  in  development  of  a  specific 
technology  -  i.e.,  "technical  push"  -  or  a  technology  may  be  developed  and  then  a  warfighter  need  is 
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discovered  that  could  use  the  technology  -  i.e.,  a  "technology  pull."  The  word  "prototype"  was  discovered  in 
the  interviews  to  have  two  possible  meanings.  In  one  case,  such  as  in  the  process  that  DARPA  typically  uses 
or  in  the  one  the  Operational  Responsive  Space  (ORS)  program  has  used,  a  prototype  is  something  developed 
quickly  in  the  lab,  tested  in  the  lab,  and  used  in  warfighter  operations.  This  is  slightly  different  than  a 
prototype  specifically  intended  to  be  used  in  an  operational  environment,  i.e.,  as  a  planned  test  path  for  a 
program  of  record.  An  example  of  the  latter  would  be  a  fly-off  of  a  new  fighter  aircraft  prototype.  The  lane  as 
defined  herein  is  meant  to  consider  those  rapid  prototypes  that  come  out  of  a  rapid  environment  and  are  not 
necessarily  intended  to  become  part  of  a  program  of  record,  at  least  not  at  the  time  that  the  prototypes  are 
tested. 


Lastly,  this  category  may  adapt  mature 
technologies  and/or  products  for  military 
use,  or  integration  with  other  military 
systems.  The  process  is  accelerated  and 
tailored  and  includes  only  the  necessary 
steps  to  quickly  develop  and  field  a  test  - 
for  example,  the  process  may  be  to:  define 
the  solution,  acquire  parts,  build  the 
system,  test  it,  ship  it,  use  it,  and  discard  it.  From  the  perspective  of  the  warfighter,  this  would  fulfill  the 
urgent  need.  The  people  involved  in  laboratory  programs  tend  to  involve  more  researchers,  PhDs,  scientists, 
and  students. 

2.  Platform  Engineering.  This  category  consists  of  modifying  and/or  integrating  existing  technology  or 
technologies  on  top  of  existing  platforms,  with  new  interfaces  [Muffatto,  2000],  The  most  predominant 
category  discovered  during  the  RT-34  site  visits  was  some  form  of  platform  engineering,  such  as  through 
replacement,  upgrades,  fixes,  and  enhancements  to  existing  platforms.  Platform  engineering  also  includes 
repurposing  of  existing  systems,  and  possibly  modifying  them,  for  different  missions.  An  example  of  platform 
engineering  is  the  Army  Prototype  Integration  Facility  (PIF)  in  Huntsville,  AL,  whose  mission  is  to  "support 
Army  Aviation,  Missile,  DoD  and  technology  activities  in  the  development,  fabrication,  integration, 
test/qualification  of  prototype  tactical  and  ground  support  systems,  subsystems  and  components." 
(http://www.redstone.army.mil/amrdec/pif/about.html) 

Additionally,  the  PIF  has  capabilities  that  allow  for  the  manufacture  and  integration  of  unique,  difficult-to- 
procure,  and  low-rate-production  items  -  which  could  fall  into  the  category  of  Lab  Demo  per  above.  Another 
example  of  platform  engineering  is  the  Mine  Resistant  Ambush  Protected  (MRAP)  vehicles,  where  the  Army 
"approved  the  emergency  expedient  of  adding  armor  kits  to  the  existing  Humvees  because  they  could  be 
fielded  more  quickly  than  the  up-armored  Humvees."  [Lamb  et  al,  2009]  The  SE  process  for  platform 
engineering  is  focused  primarily  on  the  interfaces  and  the  integration  of  the  two  existing  technologies  and 
platforms,  because  it  already  has  information  on  the  existing  systems.  People  tend  to  be  very  creative  and 
solutions-oriented,  and  they  have  a  diverse  and  long  set  of  experiences  with  the  platforms  (or  other 
platforms)  that  they  bring  to  the  table.  A  good  commercial  example  of  this  is  the  popular  television  show, 
"Monster  Garage,"  which  takes  small  teams  with  "mechanical,  fabricating  or  modifying  expertise  to  transform 
an  existing  vehicle  into  another ...  such  as  a  school  bus  into  a  pontoon  boat." 
(http://en.wikipedia.org/wiki/Monster  Garage) 

3.  Integrated  Solutions.  This  category  focuses  on  the  rapid  creation  of  a  new  platform  or  system  through  the 
integration  of  technologies  and  systems.  The  resulting  new  solution  changes  the  original  intent  of  either  (or 
both)  the  technology  or  original  system.  The  process  of  developing  an  integrated  solution  can  occur  by 
integrating  new  technology  with  an  existing  system,  or  adding  existing  technology  to  a  new  system.  The 
process  may  also  integrate  either  new  or  existing  systems  in  a  new  way,  or  on  a  new  platform  for  a  new 
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mission  in  a  new  context.  This  category  differs  from  Major  Weapons  System  in  the  level  of  oversight  and 
levels  of  technology  risk  (usually  lower/  more  mature  technology)  and  levels  of  cost;  it  is  often  characterized 
by  major  use  of  non-developmental  items  or  Commercial  Off-The-Shelf  (COTS)  components.  This  category 
may  also  use  elements  of  lanes  1  and  2  to  develop  the  new  technology. 

4.  New  Rapid  Development.  This  category  involves  more  sophisticated  and  complex  development  like 
traditional  acquisition  programs  of  record.  Programs  of  record  are  characterized  by  items  such  as  size 
(including  dollar  amount)  and  scope,  well-defined  requirements,  risk  level,  and  mission  criticality.  This 
category  may  also  utilize  new  technologies  for  a  new  platform,  or  a  new  mission  in  a  new  context,  (such  as 
the  Integrated  Solutions  category  above).  The  Air  Force  Next  Generation  Bomber  would  fall  into  this 
category.  Often  they  include  greater  amounts  of  technology  development,  manufacturing  and  production 
than  the  other  three  categories. 


3.2  Other  Lane  Taxonomies 

It  is  interesting  to  note  that  identification  and  management  of  lanes  of  acquisition  is  a  common  practice.  Readers 
should  be  familiar  with  major  DoD  acquisition  categories  (ACAT),  based  on  R&D  and  Production  dollar  thresholds. 
Other  taxonomies  include  the  NASA  risk  classes  and  Operationally  Responsive  Space  (ORS)  capability  tiers.  DoD 
bases  its  ACAT  level  only  on  dollars.  NASA  attempts  to  capture  overall  risk  and  ORS  is  more  concerned  with 
schedule,  i.e.,  time  to  deploy  a  space  capability. 

Operationally  Responsive  Space  (ORS).  ORS  defined  three  tiers  of  achieving  rapid  space  capability  as  follows. 
These  tiers  are  defined  by  the  length  of  time  it  takes  to  meet  urgent  operational  needs,  as  requested  by  the  Joint 
Force  Commanders  (JFC).  See:  http://ors.csd.disa.mil/mission/index.html  and  http://ors.csd.disa.mil/tier- 
1/index.html. 

•  Tier-1,  "Employ,"  focuses  on  existing,  on-station  capabilities.  This  first  and  most  effective  path  for 
providing  highly  responsive  effects  is  through  the  employment  of  existing,  fielded,  space  capabilities  that 
could  be  extended,  expanded,  or  changed  beyond  their  original  purpose  to  immediately  apply  to  the 
urgent  warfighter  need.  Capabilities  could  be  delivered  within  minutes  to  hours.  An  example  would  be 
moving  an  existing  satellite  to  a  new  location  to  immediately  provide  situational  data  to  warfighters. 

•  Tier-2,  "Deploy,"  is  the  fielding  of  space-ready  Capabilities.  The  focus  of  activities  in  Tier-2  solutions  is  on 
utilization  of  field-ready  capabilities  or  deploying  new  or  additional  capabilities  that  are  field  ready.  This 
involves  "responsive  exploitation,  augmentation,  or  reconstitution  of  space  force  enhancement  or  space 
control  capabilities  through  rapid  assembly,  integration,  testing,  and  deployment  of  a  small,  low-cost 
satellite."  The  goal  is  to  responsively  achieve  operational  status  for  a  newly  fielded  capability,  through  the 
process  of  rapid  deployment.  The  targeted  timeframe  for  Tier-2  solutions  is  days  to  weeks  from  the  time 
at  which  an  urgent  need  is  expressed.  ORS  developed  a  Rapid  Response  Space  Center,  initially  collocated 
at  Kirtland  Air  Force  Base  with  existing  satellite  assembly  and  test  buildings,  to  rapidly  execute  such 
capabilities. 

•  Tier-3,  "Develop,"  is  the  development  of  new  capabilities.  The  focus  is  on  urgent  needs  that  cannot  be 
addressed  with  existing  capabilities  (Tier-1)  or  through  the  rapid  deployment  of  field-ready  capabilities 
(Tier-2).  Once  developed,  these  capabilities  would  be  considered  just  like  a  "Tier  2"  capability,  and  would 
be  responsively  deployed  and  employed  in  support  of  the  requesting  warfighter  or  other  user.  The 
targeted  timeframe  for  execution  of  Tier-3  approaches  is  months  to  one  year,  recognizing  that  achieving 
such  a  challenging  timeline  requires  limiting  the  amount  of  new  development  involved. 

Tiers  1  and  2  of  the  ORS  categories  correlate  most  closely  to  the  RT-34  "Platform  Engineering"  lane  of  acquisition, 
and  Tier  3  could  fall  into  either  Lab/Prototype  or  Integrated  Solution. 
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Department  of  Defense  Acquisition  Categories  (ACAT).  Table  3-1  defines  the  DoD  acquisition  category  and  the 
reason  for  its  designation.  The  ACAT  levels  are  based  on  dollars,  with  an  implication  that  complexity  and  risk  are 
synonymous  with  cost.  The  Milestone  Decision  Authority  (MDA)  can  also  designate  a  program  as  special  interest. 
Given  the  thresholds  seen  in  the  RT-34  interviews,  rapid  programs  would  predominantly  fall  into  the  ACAT  III 
category. 


Table  3-1:  Categories  of  DoD  Acquisition  (DoD,  2008) 


Acquisition 

Category 

Reason  for  ACAT  Designation 

ACAT  1 

•  MDAP  (section  2430  of  Reference  (k)) 

•  Dollar  value:  estimated  by  the  USD  (AT&L)  to  require  an  eventual  total  expenditure  for  research, 
development,  test  and  evaluation  (RDT&E)  of  more  than  $365  million  in  fiscal  year  (FY)  2000 
constant  dollars  or,  for  procurement,  of  more  than  $2,190  billion  in  FY  2000  constant  dollars 

o  MDA  designation 

•  MDA  designation  as  special  interest 

ACAT  IA  1,2 

•  MAIS  (Chapter  144A  of  Reference  (k)) :  A  DoD  acquisition  program  for  an  Automated  Information 
System  3  (either  as  a  product  or  a  service)  that  is  either: 

o  Designated  by  the  MDA  as  a  MAIS;  or 
o  Estimated  to  exceed: 

■  $32  million  in  FY  2000  constant  dollars  for  all  expenditures,  for  all  increments,  regardless  of  the 
appropriation  or  fund  source,  directly  related  to  the  AIS  definition,  design,  development,  and 
deployment,  and  incurred  in  any  single  fiscal  year,  or 

■  $126  million  in  FY  2000  constant  dollars  for  all  expenditures,  for  all  increments,  regardless  of  the 
appropriations  or  fund  source,  directly  related  to  the  AIS  definition,  design,  development,  and 
incurred  from  the  beginning  of  the  Materiel  Solution  Analysis  Phase  through  deployment  at  all  sites; 

or 

■  $378  million  in  FY  2000  constant  dollars  for  all  expenditures,  for  all  increments,  regardless  of  the 
appropriation  or  fund  source,  directly  related  to  the  AIS  definition,  design,  development, 
deployment,  operations  and  maintenance,  and  incurred  from  the  beginning  of  the  Material  Solution 
Analysis  Phase  through  sustainment  for  the  estimated  useful  life  of  the  system. 

•  MDA  designation  as  a  special  interest 

ACAT  II 

•  Does  not  meet  criteria  for  ACAT  1 

•  Major  system 

o  Dollar  value:  estimated  by  the  DoD  Component  Flead  to  require  an  eventual  total  expenditure  for 
RDT&E  of  more  than  $140  million  in  FY  2000  constant  dollars,  or  for  procurement  of  more  than 
$660  million  in  FY  2000  constant  dollars  (section  2302d  of  Reference  (k)) 
o  MDA  designation  4  (paragraph  (5)  of  section  230  of  Reference  (k)) 

ACAT  III 

•  Does  not  meet  criteria  for  ACAT  II  or  above 

•  AIS  that  is  not  a  MAIS 

NASA.  The  National  Aeronautics  and  Space  Administration  (NASA)  utilizes  four  "Classes"  of  missions,  defined  in 
NPR  8705.4  -  Risk  Classification  for  NASA  Payloads.  These  classes  incorporate  aspects  of  mission  priority,  risk, 
national  interest,  complexity  and  cost.  See  Table  3-2. 
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Table  3-2:  Classes  of  NASA  Systems 


Characterization 

Class  A 

Class  B 

Class  C 

Class  D 

Priority  (Criticality  to 

Agency  Strategic  Plan)  and 
Acceptable  Risk  Level 

High  priority,  very 
low  (minimized) 
risk 

High  priority,  low 
risk 

Medium  priority, 
medium  risk 

Low  priority,  high 
risk 

National  Significance 

Very  high 

High 

Medium 

Low  to  Medium 

Complexity 

Very  high  to  high 

High  to  medium 

Medium  to  low 

Medium  to  low 

Mission  Lifetime  (Primary 
Baseline  Mission) 

Long,  >5  years 

Medium,  2-5  years, 

Short,  <2  years 

Short,  <2  years 

Cost 

High 

High  to  medium 

Medium  to  low 

Low 

Launch  Constraints 

Critical 

Medium 

Few 

Few  to  none 

In-Flight  Maintenance 

N/A 

Not  feasible  or 

difficult 

Maybe  feasible 

May  be  feasible  and 
planned 

Alternative  Research 
Opportunities  or  Re-flight 
Opportunities 

No  alternative  or 

re-flight 

opportunities 

Few  or  no 

alternative  or  re¬ 
flight  opportunities 

Some  or  few 

alternative  or  re¬ 
flight  opportunities 

Significant 
alternative  or  re¬ 
flight  opportunities 

Achievement  of  Mission 

Success  Criteria 

All  practical 

measures  are 

taken  to  achieve 

minimum  risk  to 

mission  success. 

The  highest 

assurance 

standards  are 

used 

Stringent  assurance 
standards  with  only 
minor  compromises 
in  application  to 
maintain  a  low  risk 

to  mission  success 

Medium  risk  of  not 
achieving  mission 
success  may  be 
acceptable.  Reduced 
assurance  standards 
are  permitted 

Medium  or 
significant  risk  of  not 
achieving  mission 
success  is  permitted. 
Minimal  assurance 

standards  are 
permitted 

http://nodis3.gsfc.nasa.gov/displayDir.cfm7lnternal  ID=N  PR  8705  0004  &page  name=AppendixA 


A  related  approach  is  used  by  NASA  for  launch  vehicles,  documented  in  their  Launch  Services  Risk  Mitigation 
Policy.  Similarly,  three  classes  (and  the  corresponding  mission  Classes)  are  used  to  assess  both  the  cost  and  the 
attributes  of  the  mission.  Table  4.3  reflects  the  attributes  of  management,  launch  flight  test,  manufacturing 
audits,  T&E  planning,  quality,  and  safety.  Other  attributes  include  risk. 

Table  3-3:  Classes  of  Launch  (NASA) 


Launch  Vehicle 
Risk  Category 

Category  1  (High  Risk) 

Category  2  (Medium  Risk) 

Category  3  (Low  Risk) 

Payload  Class  (per 

D 

C&D,  sometimes  B 

A,  B,  C  &  D 

NPR  8705.4) 

Alternative  1 

Alternative  2 

Alternative  1 

Alternative  2 

Alternative  3 

Management 

AS9100  or  ISO  9001 

AS9100  Complaint 

AS9100  Complaint 

AS9100 

AS9100 

AS9100 

Systems 

Compliant 

Complaint 

Compliant 

Compliant 

Flight  Experience 

No  previous  flights 

1  successful  flight 

3  (minimum  2 

14  consecutive 

6  successful 

3  (minimum  2 

required,  can  use  the 

of  a  common 

consecutive) 

successful 

flights  (minimum 

consecutive) 

first  flight  of  a  common 

launch  vehicle 

successful  flights 

flights  (95% 

3  consecutive)  of 

successful 

launch  vehicle 

configuration, 

of  a  common 

demonstrated 

a  common 

flights  of  a 

configuration, 

instrumented  to 

launch  vehicle 

reliability  at 

launch  vehicle 

common 

instrumented  to 

provide  design 

configuration, 

50% 

configuration, 

launch  vehicle 

provide  design 

verification  &  flight 

instrumented  to 

confidence)  of 

instrumented  to 

configuration, 

verification  &  flight 

performance  data 

provide  design 

a  common 

provide  design 

instrumented 

performance  data 

verification  & 

launch  vehicle 

verification  and 

to  provide 

Post-Flight 

flight  performance 

configuration, 

flight 

design 

Post-Flight 

Operations/Anomal 

data 

instrumented 

performance 

verification  & 

Operations/Anomaly 

y  Resolution 

to  provide 

data 

flight 

Resolution  Process 

Process 

Post-Flight 

design 

performance 

Operations/Anom 

verification  and 

Post-Flight 

data 

Flight  Data  Assessment 

NASA  Flight  Margin 

aly  Resolution 

flight 

Operations/Ano 

Process 

Verification 

Process 

performance 

maly  Resolution 

Post-Flight 
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NASA  Flight 

Margin 

Verification 

data 

Post-Flight 

Operations/An 

omaly 

Resolution 

Process 

NASA  Flight 

Margin 

Verification 

Process 

NASA  Flight 

Margin 

Verification 

Operations/An 

omaly 

Resolution 

Process 

NASA  Flight 

Margin 

Verification 

Design 

NASA  assessment  of 

LSC  design  reliability 

NASA  assessment 
of  LSC  design 
reliability 

NASA  assessment 
of  LSC  design 
reliability 

NASA 

assessment  of 

LSC  design 
reliability 

NASA  assessment 
of  LSC  design 
reliability 

NASA 

assessment  of 

LSC  design 
reliability 

Mfg  &  Ops  and 
Systems 

Engineering 

NASA  Audits 

Documented  ICD 

Process 

NASA  Audits 

NASA  Audits 

None 

NASA  Audits 

NASA  Audits 

System  Safety 

FMEA  for  all  safety 
critical  component 

Preliminary  &  Final 

Hazard  Analysis 

Compliance  with 
applicable  Range  Safety 
Requirements 

Demonstrated 
compliance  with 
applicable  Range 
Safety 

Requirements 

Demonstrated 
compliance  with 
applicable  Range 
Safety 

Requirements 

Demonstrated 
compliance 
with  applicable 
Range  Safety 
Requirements 

Demonstrated 
compliance  with 
applicable  Range 
Safety 

Requirements 

Demonstrated 
compliance 
with  applicable 
Range  Safety 
Requirements 

Test  &  Verification 

Acceptance  Test  Plan  in 
place 

Ground  Test,  End-to- 
End  Tests  complete 

Comprehensive 
Acceptance  Test 
results 

NASA  Design 
Certification 

Review 

None 

NASA  Design 
Certification 

Review 

Comprehensive 

Acceptance 

Test  results 

Quality 

Systems/Process 

NASA  Audit 

NASA  Audit 

NASA  Audit 

None 

NASA  Design 
Certification 

Review 

Comprehensive 

Acceptance 

Test  results 

Flight  Hardware  & 

Software 

Qualification 

Qualified  Hardware  (for 
space  application) 

Testing  completed 

Series  of  NASA 
Engineering  Review 
Boards  on  vehicle 
subsystems 

NASA  Design 
Certification 

Review 

None 

NASA  Design 
Certification 

Review 

Series  of  NASA 
Engineering 
Review  Boards 

on  vehicle 
subsystems 

LV  Analysis 

Analysis  Plan/Definition 

Analysis 

Plan/Definition 

NASA  Coupled 

Loads  Analysis 

iv&v 

NASA  IV&V 

None 

NASA  IV&V 

NASA  IV&V 

Risk  Management 

Risk  Plan,  Mitigated  and 
Accepted  Technical  and 
Safety  Risks 

Risk  Plan,  Mitigated 
and  Accepted 
Technical  and 

Safety  Risks 

Risk  Plan, 

Mitigated  and 
Accepted 

Technical  and 

Safety  Risks 

Risk  Plan, 
Mitigated  and 
Accepted 
Technical  and 
Safety  Risks 

Risk  Plan, 

Mitigated  and 
Accepted 

Technical  and 
Safety  Risks 

Risk  Plan, 
Mitigated  and 
Accepted 
Technical  and 
Safety  Risks 

Integrated 

Analysis 

None 

None 

None 

None 

None 

Full  Vehicle 

Fishbone 

Launch  Complex 

None 

NASA  Engineering 
Review  Board 

NASA  Design 
Certification 

Review 

None 

NASA  Design 
Certification 

Review 

NASA 

Engineering 
Review  Board 

http://nodis3.gsfc.nasa.gov/NPD  attachments/N  PD  8610  007D  A. pdf 


Observations.  There  are  interesting  connotations  and  observations  by  examining  these  rapid  lanes.  Lane  et  al 
{{189  Lane,  J.  2010}}  in  their  study  of  rapid  acquisition  success  factors,  offered  that  the  processes  and  personality 
types  may  also  differ  across  types  of  rapid  environments.  Laboratory  prototyping  may  have  less  structure,  more 
innovation  and  trial-error  experimentation.  Integration  solutions  may  have  more  formal  processes  with  greater 
lifecycle  considerations  (and  the  people  to  consider  them).  This  is  further  reflected  in  Finding  4.2. 
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3.3  Flexibility  Exists  Within  and  Throughout  All  Lanes 

Maybe  this  finding  is  not  surprising  -  there  is  huge  flexibility  both  within  and  across  these  lanes  of  acquisition. 
This  flexibility  involves  people  (hiring)  and  process. 


3.3.1  Flexibility  in  Hiring  the  Right  People 

There  was  an  overwhelming  response  from  the  interviews  on  the  importance  of  people  in  expedited  SE  and 
urgent  needs  projects.  For  example,  the  right  people  could  execute  any  process;  but  a  process  alone  without  the 
right  people  does  not  guarantee  success.  Some  of  the  key  comments  from  interviews  include  the  desire  for  small, 
handpicked  teams;  selection  of  the  "A"  team  or  "a  team";  and  ensuring  a  mix  of  diverse  experiences. 

One  of  the  first  site  visits  was  with  an  Air  Staff  rapid  acquisition  office.  They  were  highly  selective  in  bringing  in 
civilians  and  military  into  this  organization.  It  was  mentioned  that,  "they  interview  100  to  hire  a  few."  Generally 
this  is  a  unique  situation  for  large  program  offices,  where  personnel,  both  military  and  civilian  get  assigned 
through  normal  rotations.  Another  organization  mentioned  the  use  of  "recommendations."  They,  too,  were 
highly  selective  and  if  a  trusted  member  (such  as  a  past  employee)  recommended  someone,  that 
recommendation  was  as  good  as  an  interview. 

Another  aspect  that  was  important  regarding  personnel  selection  was  personality  type.  In  almost  every  case, 
these  lean,  agile,  rapid  organizations  were  looking  for  highly  motivated  and  competent  government  employees. 
These  people  were  comfortable  with  uncertainty  and  flexibility  of  process.  Interesting  to  note,  not  all  of  these 
rapid  team  members  can  be  same.  One  organization  mentioned  they  had  "5  Tiggers  and  1  Eeyore"  (reminiscent 
of  the  characters  in  "Winnie  the  Pooh").  You  need  to  have  a  "naysayer"  to  ensure  you  don't  run  too  rapid, 
without  missed  critical  considerations. 

Another  organization,  as  mentioned  in  Group  2,  previously  had  the  ability  to  selectively  choose  personnel,  but 
now  receives  personnel  through  the  normal  rotation.  As  a  result,  they  were  no  longer  able  to  pre-select 
personnel  according  to  specific  skill  needs  or  personalities.  Instead  of  fretting  about  this  change,  the  organization 
instead  changed  its  strategy  to  where  it  annually  evaluated  the  mix  of  personnel,  based  on  their  competencies, 
skills,  experiences,  and  desires.  Then  they  defined  the  portfolio  of  work  the  organization  was  capable  of 
achieving  in  a  given  year,  based  on  that  mix  of  personnel.  This  was  a  very  innovative  way  of  making  real-time 
adjustments  to  capability  based  on  personnel. 

Regardless  of  who  was  selected  or  desired  for  an  organization,  military  organizations  have  the  further  challenge  of 
adjusting  head  count  and  skill  mix  when  personnel  are  deployed.  This  often  leaves  gaps  that  cannot  be  filled 
easily. 


3.3.2  Flexibility  in  Process 

Process  variation  was  another  area  that  was  used  to  achieve  flexibility  -  especially  in  contracting  -  but  also  in 
management  and  SE.  The  theme  of  flexibility  arose  repeatedly  in  the  SME  discussions,  with  organizations  often 
detailing  the  contracting  approach  they  used  and  how  this  facilitated  (or  hindered)  the  ability  to  execute  rapidly. 
The  application  of  flexibility  was  also  dependent  on  the  experiences  of  the  technical  personnel  involved. 
Flexibility  and  risk  go  hand  in  hand,  so  it  is  important  to  manage  both. 
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For  example,  one  organization  had  a  several  year  IDIQ  contract  to  rapidly  add  tasks  for  local  on-site  design  and 
production  services.  These  included  welding,  machining,  cable  assembly  and  checkout,  breadboard  fabrication, 
install,  test  etc.  Interestingly,  these  services  were  performed  in  government-owned,  government  operated 
(GOGO)  facilities. 

Other  contracting  approaches,  used  by  this  organization  and  several  others,  including  classified  organizations, 
provided  access  to  subject  matter  expertise  through  several  sole-source  contracts.  This  allowed  immediate  access 
to  a  pool  of  uniquely  qualified  and  pre-identified  set  of  individuals  that  could  be  rapidly  requested  to  perform 
work  as  needed.  Competition  was  also  used,  depending  on  the  type  of  work  involved  (e.g.,  execution  of  a  build  to 
print  item),  the  expertise  required  (e.g.,  was  it  unique  or  were  there  multiple  qualified  sources),  and  the 
personnel  and  contractors  involved.  Even  when  competition  was  used,  the  entire  program  understood  the  need 
to  proceed  quickly,  such  that  award  and  delivery  of  weapon  systems  could  be  done  in  several  months  (not  years). 

The  choice  of  contracting  instrument  was  often  based  on  the  experience,  preference,  and  "comfort  level"  of  the 
contracting  officer.  Contracting  officers  who  were  embedded  in  rapid  organizations  typically  had  the  most 
knowledge  of  the  flexibility  available  in  existing  DoD  policy  and  Federal  Acquisition  Regulations  (FARs).  But  most 
often,  flexible  contracting  practices  are  not  well-known,  trusted,  understood  or  utilized  because  they  are  not  the 
approaches  used  in  traditional  acquisition  -  or  they  may  not  be  appropriate  -  and  therefore  contracting  personnel 
are  not  familiar,  nor  comfortable  with  working  with  new  and  different  mechanisms.  Examples  of  flexible 
contracting  mechanisms  include:  fixed  price  contracts,  IDIQs,  sole  source,  commercial  terms  and  conditions, 
waivers  (such  as  for  certified  cost  and  pricing  data),  FAR  Part  12  commercial  tailored  contracts,  Other  T ransaction 
Authority  (OTA),  and  NASA  Space  Act  Agreements  (SAAs,  i.e.,  NASA's  form  of  an  OTA). 

DARPA  uses  "Other  Transaction  Authority"  in  its  rapid  development  programs.  OTAs  are  fixed  price,  milestone 
based  contracts,  with  no  FARs  and  only  the  minimum  statutory  terms  included.  This  is  a  very  friendly  way  to 
rapidly  get  on  contract,  especially  for  small  businesses,  entrepreneurs,  or  other  non-traditional  contractors.  The 
Missile  Defense  Agency  (MDA)  also  has  OTA  authority.  Another  mechanism  is  FAR  Part  12  commercial  tailored 
contracts.  "FAR  Part  12"  is  used  to  procure  commercial  items  on  a  fixed  price  basis.  In  an  attempt  to  educate 
contracting  and  personnel  about  how  to  use  FAR  Part  12,  a  "Consistency  in  the  Acquisition  of  Commercial  Items," 
memo  was  issued  by  the  Undersecretary  of  Defense  for  Acquisition,  Technology  &  Logistics  [USD  (AT&L)]  in  July 
2001,  followed  by  a  "Commercial  Item  Handbook,"  in  November  2001. 

NASA  has  more  recently  used  contractual  flexibility  with  the  activities  involved  in  commercial  cargo  and  crew  for 
the  International  Space  Station  (ISS).  SAAs  were  used  in  the  initial  development  process,  with  highly  tailored 
management,  engineering,  and  test  processes.  The  intent  is  to  use  FAR  Part  12  to  procure  the  actual  commercial 
services  from  commercial  entities. 

The  RT-34  discussions  also  revealed  tailoring  and  scaling  processes  for  systems  engineering  and  program 
management,  depending  on  what  product  was  being  developed  (i.e.,  what  the  research  came  to  call  the  lane  of 
acquisition)  and  who  was  working  on  the  program  (i.e.,  the  level  of  experience,  capabilities,  and  competencies  of 
the  individuals  and  the  combined  team).  The  results  show  that  rapid  organizations  first  defined  what  it  is  they  are 
trying  to  accomplish,  and  then  they  define  the  scope  of  the  effort  to  match  the  time  available.  The  team 
determines  the  "most  important"  SE  and  program  management  artifacts  needed,  and  develop  a  program  plan 
with  a  timeline  and  milestones  that  match  the  level  of  effort.  As  seen  in  one  organization,  this  process  was 
described  as  a  "mini,  compressed"  equivalent  of  a  DoD  5000  series,  scaled  down  to  only  the  necessary  milestones 
and  minimal  reviews  appropriate  for  that  particular  urgent  need.  The  focus  was  on  execution,  and  the  minimum 
amount  of  paperwork,  meetings,  and  oversight  to  accomplish  the  task. 
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The  ability  to  tailor  in  this  way  may  depend  on  the  organization  using  elements  of  the  Level  2  framework,  such  as 
"intense  knowledge  sharing"  (e.g.,  frequent  communications  to  keep  the  team  on  the  same  page  at  all  time 
without  artificial  reviews  conducted  for  review  sake,  or  embedding  the  warfighter  with  the  developer)  and  a  risk- 
based  culture.  An  example  of  the  latter  is  that  the  ability  to  tailor  requires  the  judgment  calls  of  experienced 
personnel,  who  have  the  scars  to  show  for  many  previous  programs  and  scenarios,  which  is  then  used  to  make 
decisions  and  evaluate  risk  at  any  given  time  in  the  program. 


3.4  Enablers  to  Group  2 

Similar  to  Groups  1  and  3,  the  Rapid  Best  Practices  observed  in  Group  2  may  also  be  facilitated  by  new  tools  and 
information  technology.  Tools  that  help  intensive  knowledge  sharing  will  facilitate  real-time  management.  Other 
tools,  such  as  the  COCOMO  parametric  suite  of  cost  and  schedule  estimator,  can  be  used  throughout  the  rapid 
lifecycle.  See  Appendix  D  for  further  details. 
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4  Inferred  Organizational  Characteristics  for  "Go  Fast"  Cultural  Best  Practices  (Group  3) 


In  this  chapter,  we  inferred  additional  findings  from  the  interviews  and  literature  review  that  are  specifically 
focused  on  expedited  processes  supported  from  across  the  enterprise.  We  refer  to  these  as  "go  fast"  cultural 
best  practices.  Here  is  where  rapid  organizations  really  start  to  "go  fast"  and  differ  from  traditional  acquisition 
programs.  There  are  cultural  shifts  in  the  organization  from  top  to  bottom.  From  this  analysis,  we  offer  3 
findings.  The  findings  include  knowledge  sharing,  a  risk-focused  culture,  and  the  organization's  ambidexterity 
(plan/execute)  abilities.  It  also  includes  an  ability  to  employ  real-time  management,  from  top  to  bottom.  These 
"go  fast"  concepts  are  all  cultural  best  practices  that  traverse  work  units,  functional  partitions  and  project  groups. 

The  Inferred  Characteristics  are: 

•  Intense  and  efficient  knowledge-sharing  is  used  to  enable  stabilization  and  synchronization  of  information 

•  Rapid  organizations  are  characterized  by  a  risk  culture 

•  Rapid  organizations  are  structured  for  ambidexterity 

•  Rapid  DoD  acquisition  employs  real-time  management. 


4.1  Intense  and  Efficient  Knowledge-Sharing  is  Used  to  Enable  Stabilization  and  Synchronization  of  Information 

Rapid  organizations  repeatedly  mentioned  the  need  for  intense  and  efficient  knowledge-sharing.  The  knowledge 
sharing  had  a  particular  purpose:  for  synchronization  and  stabilization  of  information.  Synchronization  was 
needed  because,  in  a  rapid  work  environment,  information  is  often  entering  the  program  or  project  at  such  a 
speed  to  different  personnel  that  it  is  necessary  to  determine  if  everyone  knows  the  same  information,  and  if  they 
don't,  are  caught  up.  Otherwise,  decisions  will  be  made  based  on  obsolete  or  incomplete  information. 
Stabilization  of  information  is  also  critical,  particularly  prior  to  decision-making  -  even  if  the  stabilization  is 
temporary  -  so  that  the  evidence  for  the  decision  is  clarified,  vetted,  and  then  appropriately  used  in  the  short 
time  available. 

Such  continuous  and  intense  knowledge-sharing  could  have  the  adverse  effect  of  slowing  rapid  teams.  However, 
in  our  discussions,  the  organizations  found  ways  to  make  the  intense  knowledge  sharing  as  efficient  as  possible. 
Several  of  the  interviewees  mentioned  that  collocation  was  helpful  for  fast  knowledge-sharing  because  it  allowed 
people  to  easily  inform  others  spontaneously  of  new  information.  In  fact,  collocation  was  mentioned  thirty  times. 
This  enabled  everyone  to  have  access  to  the  project,  with  constant  updates  on  all  project  areas,  and  facilitated 
constant  integration  of  those  different  project  areas.  One  organization  in  particular  would  bring  customers  on-site 
during  the  requirements  development  phase  to  speed  up  the  requirements  phase  and  get  a  contract  in  "20  days 
vs  1  year." 

In  these  cases,  collocation  was  also  used  as  a  way  to  interface  with  the  user  as  early  and  often  as  possible.  When 
the  warfighter  was  collocated  with  the  design  team,  even  for  a  partial  time,  the  team  was  able  to  more  quickly 
discern  what  the  warfighter  actually  needed  and  do  rapid  trades  to  get  to  an  efficient  solution. 

Collocation,  however,  is  impractical  in  those  many  situations  in  which  the  group  is  larger,  temporary,  distributed, 
or  involves  members  who  travel  a  fair  amount.  Moreover,  intensely  sharing  information  via  collocation  also  leads 
to  significant  disruptions,  making  it  more  difficult  for  individuals  to  complete  the  task  at  hand.  While  alternatives 
to  collocation  were  not  often  mentioned,  in  one  case,  a  project  management  dashboard  was  used  to  efficiently 
share  knowledge  to  all  program  members  on  the  status  of  the  projects,  and  in  another  case,  a  new  chatroom  had 
been  started  to  facilitate  knowledge-sharing. 
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Collocation  also  may  not  be  needed  if  the  context  of  the  work  and  the  construct  of  the  team  can  achieve  the  same 
result.  For  example,  one  commercial  software  team  that  practiced  agile  software  development  explained  that 
collocation  was  an  insult  to  them.  The  design  team  had  worked  together  for  25  years  and  each  knew  all  elements 
of  the  legacy  software.  They  thrived  working  under  their  own  work  habits  -  one  liked  to  work  early,  another 
stayed  up  all  night,  a  third  liked  to  work  from  home.  The  closeness  of  the  team  and  their  knowledge  of  the 
product  and  of  each  other  meant  that  they  had  built  extreme  trust  and  they  had  different  ways  of  achieving  the 
knowledge  transfer  that  collocation  is  often  sought  to  provide. 

We  observed  that  collocation  was  one  example  of  "how"  organizations  rapidly  shared  information  and  other 
techniques  including  "digitally  enabled"  tools,  as  further  explored  in  Appendix  D.  The  bottom  line  is  that 
organizations  recognized  the  need  to  have  constant  and  equal  access  to  information  and  the  ability  to  turn  this 
into  usable  knowledge;  how  they  did  this  varied,  and  may  depend  on  the  acquisition  lane,  circumstance, 
environment,  culture,  or  access  to  information  technology  tools. 


4.2  Rapid  Organizations  are  Characterized  by  Risk-Focused  Culture 

Consistent  across  the  rapid  organizations  was  a  characteristic  that  we  have  labeled:  "a  risk  focused  culture."  By 
culture,  we  refer  to  the  norms  that  permeated  the  organization  and  the  overarching  focus  of  personnel  as  they 
went  about  their  work.  We  refer  to  this  culture  as  "risk-focused"  because  it  was  one  of  confronting,  identifying, 
and  understanding  risk,  deciding  to  accept  it,  mitigate  it  or  remove  it,  then  moving  forward  and  monitoring  the 
risk.  There  was  a  constant  awareness  of  the  risks  involved.  In  fact,  risks  were  seen  as  normal  aspects  of  their 
work.  In  rapid  cultures,  each  person's  job  has  risk  components  to  it,  and  each  person  is  expected  to  exercise 
appropriate  discretion  in  confronting,  identifying,  sharing,  mitigating,  and  accepting  the  risks.  For  example,  a 
financial  contracts  officer  may  need  to  accept  some  risk  in  moving  quickly  to  resolution  on  getting  money 
creatively  to  the  team.  The  rapid  organizations,  then,  were  not  trying  to  eliminate  all  risks,  since  there  seemed  to 
be  recognition  that  such  an  attempt  would  be  foolhardy.  Instead,  they  were  accepting  of  riskiness  of  what  their 
work  involved  and  understood  the  context  of  the  risk. 

Accepting  risk  did  not  mean  inaction  or  fear.  To  the  contrary,  accepting  risk  meant  that  personnel  in  rapid 
organizations  were  constantly  engaged  in  thinking  through  contingency  plans,  identifying  and  exploring  root 
causes  of  the  risks  in  hopes  of  diverting  the  risk,  and  monitoring  the  risks  so  that  mitigation  actions  could  be  taken 
if  need  be.  In  some  cases,  it  was  enough  to  be  aware  of  and  accept  (take)  the  risk. 


For  example,  one  organization  developing  prototypes  found  their 
prototype  had  an  immediate  application  for  a  warfighter  and  could 
be  used  in  the  field  for  an  operational  purpose  earlier  than  expected. 
Flowever,  there  was  no  time  to  include  a  safety  feature  that  was 
planned  for  later  in  the  development  process.  The  risk  of  sending 
this  prototype  in  the  field  without  the  feature  was  mitigated  because 
the  particular  warfighter  was  experienced,  made  aware  of  the  risk, 
and  understood  how  to  use  the  prototype  without  that  feature.  This 
risk  was  worth  taking.  The  team  (consisting  of  the  rapid  organization 
and  the  end  user  warfighter)  mutually  agreed  to  send  the  product 
into  the  field  without  the  feature.  On  the  other  hand,  had  the 
warfighter  been  inexperienced  or  not  knowledgeable  of  the  risks 
involved  with  using  a  system  without  this  feature  (such  as  an  18-year 
old  soldier  fresh  out  of  boot  camp)  then  it  may  not  have  been  appropriate  to  send  the  prototype  into  the  field  at 
that  time. 


Dedicated  Contracting  Office 


•  One  organization  cited  importance  of 
dedicated  contracting  division 

•  Otherwise,  rely  on  base  contracting  and 
hope  to  get  the  same  person  on  the 
phone  each  time 

•  Flave  them  set  everything  aside 

•  Keep  paperwork  in  project  folder 

•  Speed  contracting  process-10  days 
instead  of  11 

•  Not  rapid,  but  faster 
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Research  on  risk-focused  cultures  in  industry  has  repeatedly  demonstrated  that  they  are  only  possible  when 
management  fosters  a  "climate  of  psychological  safety"  (Edmondson  et  al  2001).  A  climate  of  psychological  safety 
involves  leadership  offering  "topcover"  (i.e.,  supporting  the  personnel  when  criticized  by  external  parties)  to  their 
personnel  as  well  as  management  expectations  backed  by  an  incentive  reward  structure  for  personnel  to  point 
out  risks,  take  risks,  accept  risks,  and  mitigate  risks  with  the  expectation  to  understand  and  be  fully  aware  and 
transparent  of  what  risks  are  where  and  why.  In  other  words,  the  reward  system  must  also  match  the  risk  culture. 
Personnel  do  not  get  fired  for  taking  a  risk  or  making  a  mistake.  This  culture  is  created  when  learning,  story¬ 
telling,  and  mentoring  are  encouraged  and  practiced  and  both  successes  and  failures  are  shared  such  as  to  avoid 
making  the  same  mistakes  twice. 


4.3  Rapid  Organization  are  Structured  for  Ambidexterity 

In  corporate  contexts,  we  are  increasingly  observing  the  presence  of  what  are  referred  to  as  "ambidextrous 
organizations"  (Birkinshaw  and  Gibson  2004).  Ambidextrous  organizations  have  two  different  structures  in  place: 
one  structure  focuses  on  exploration,  and  another  that  focuses  on  exploitation.  The  structure  focused  on 
exploitation  generally  has  substantial  routines  in  place,  including  milestones,  project  management  practices,  and 
specific  work  activities  identified  that  should  be  performed  in  particular  order.  Personnel  working  in  exploitation 
structures  tend  to  be  rewarded  for  knowing  the  rules  and  following  standard  processes.  The  expectation 
conveyed  by  management  to  personnel  working  in  an  exploitation  structure  is  take  the  product  and  process  as 
given,  with  a  focus  on  "efficiently  executing  to  plan". 

In  contrast  to  the  exploitation  structure,  the  structure  focused  on  exploration  generally  encourages 
entrepreneurial  spirit,  innovation,  experimentation,  learning,  iteration,  and  risk.  The  expectation  conveyed  by 
management  to  personnel  working  in  an  exploration  structure  is  that  the  product  or  process  is  not  given,  but 
needs  to  be  changed  in  ways  not  initially  anticipated.  Personnel  working  in  exploration  structures  tend  to  be 
rewarded  for  taking  risks,  generating  new  information  or  technologies,  and  creating  new  opportunities  for  the 
program.  Such  personnel  tend  to  be  explorers  themselves,  comfortable  with  ambiguity,  uncertainty,  and 
challenges  that  may  or  may  not  be  known  or  resolvable. 

In  a  corporate  context,  exploration  structures  may  describe  the  early 
concept  development  phase,  while  exploitation  structures  may  describe 
the  implementation  or  operational  phase  of  an  activity.  However,  in  a 
truly  ambidextrous  organization,  both  structures  are  able  to  be  invoked 
at  any  point  in  time  depending  on  the  needs  of  the  customer  or  new 
information  received  from  outside  the  organization  that  suggests  a 
requirement  to  switch  from  exploration  to  exploitation  or  visa-versa.  For 
an  ambidextrous  organization  to  be  able  to  switch  between  exploration 
and  exploitation,  personnel  and  routines  need  to  be  in  a  place  that  they 
are  able  to  know  when  to  switch  and  how  to  switch.  An  ambidextrous 
organization,  then,  will  separate  personnel  by  those  who  are  comfortable  in  exploration  modes  and  those  who 
are  comfortable  in  exploitation  modes,  and  will  also  have  personnel  who  are  comfortable  in  both  modes,  as  well 
as  sufficiently  experienced  and  self-confident  to  make  judgments  about  when  and  how  to  invoke  the  different 
modes.  An  example  in  corporate  contexts  is  a  product  development  manager  who  is  both  working  to  get  a 
product  out  as  well  as  working  on  new  high-risk  R&D. 

We  observed  several  of  the  rapid  organizations  exercising  this  ambidexterity.  On  the  exploration  side,  there  was 
a  constant  exploring  to  anticipate  future  need.  They  wouldn't  simply  respond  to  a  warfighter's  statement  of 
need,  but  instead  would  explore  what  the  problem  was  and  generate  new  ideas  to  help  the  warfighter  achieve 
the  end  objective.  Moreover,  the  rapid  organizations  we  interviewed  were  not  simply  exploring,  but  finding  ways 


"Multi-Lingual"  Personnel 


•  One  organization  cited  importance  of 
employees  with  broad  experience 

•  Prioritized  that  everyone  be  multi¬ 
purpose 

•  "Speak  two  or  three  things" 

•  Not  just  in  their  own  box 
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to  explore  faster.  The  exploitation  structure  existed  as  well,  as  there  were  milestones,  project  management 
practices,  and  deliverables  expected.  Finally,  the  rapid  organizations  were  able  to  switch  practices  quickly.  For 
example,  in  one  organization,  an  urgent  need  by  the  chain  of  command  quickly  converted  the  lab  project  into  a 
fielded  demo;  likewise,  an  urgent  need  often  triggers  a  variety  of  platform  modifications. 

Ambidexterity  also  is  seen  in  the  dictionary  context  of  highly  skilled,  adept  people  who  are  equally  expert  at  both 
hands.  These  multi-lingual  personnel  tend  to  have  broad  experience  in  many  areas  and  are  capable  of  working 
multiple  jobs.  Several  organizations  talked  about  the  having  people  on  their  teams  that  were  capable  of  serving  in 
multiple  roles.  For  example,  engineers  who  had  rotated  through  acquisition  jobs  had  a  greater  appreciation  and 
knowledge  of  budgeting  and  contracting.  These  were  true  systems  thinkers  who  could  solve  problems  and 
contribute  to  multiple  domains,  given  the  depth  and  breadth  of  their  experience. 


In  switching  rapidly  between  exploration  and  exploitation,  we 
observed  a  concern  expressed  by  the  managers,  that  we  describe 
"technical  debt".  A  term  from  the  agile  software  development 
community,  "technical  debt"  is  a  metaphor  for  work  or  rigor  not 
accomplished,  often  in  one  dimension  of  a  program  (like  architectural 
simplicity,  modularity  or  use  of  commercial  off  the  shelf  (COTS) 
systems,  often  to  meet  an  urgent  need  or  schedule  deadline.)  These 
compromises  often  have  to  be  paid  back,  eventually,  and  often  with 
"interest"  for  the  long-term  stability  and  success  of  a  program  that  is 
intended  to  have  a  long  life  cycle. 

As  rapid  organizations  move  quickly,  the  decisions  they  may  or  may 
not  come  back  to  haunt  them,  and  they  often  do  not  know  which  of 
those  decisions  are  the  ones  that  will  come  back.  There  may  be  areas 
where  rapid  development  takes  more  time  up  front  to  consciously  make  decisions  that  lead  to  a  system 
architecture  that  is  sustainable  in  the  long  term,  if  that  is  desirable. 


Programs  of  Record:  Just  Off  Rapid's  Flank 


•  Important  to  have  a  competent 
government  Team 

•  One  organization  employed  prior  and 
currently  enlisted  project  members 

•  Stay  close  to  user/warfighter 

•  Constant  communication 

•  Constant  focus  on  "off  ramp" 

•  Program  in  use  longer  than  a  year  rolled 
into  Programs  of  Record 

•  If  less  than  year-focus  on  rapid  fielding; 
work  out  off  ramp  later 


4.4  Rapid  DoD  Acquisition  Employs  Real-Time  Management 

A  unique  characteristic  of  the  DOD  vis-a-vis  corporate  contexts  is  the  opportunity  that  young  people  in  the 
military  are  given  to  learn,  accept  substantial  responsibility,  and  gain  experience  in  making  decisions  under 
uncertain  conditions.  The  researchers  found  in  the  site  visit  discussions,  that  the  rapid  organizations  were 
managed  in  a  manner  that  explicitly  provided  these  opportunities.  Personnel  were  empowered  to  learn, 
understand  and  accept  responsibility  for  the  risks  they  incurred,  and  were  expected  to  make  decisions  about 
acquisitions,  despite  not  having  complete  information.  While  all  DOD  contexts  encourage  empowerment  at  the 
lowest  level,  rapid  organizations  face  the  additional  challenge  that  the  empowerment  needs  to  be  done  in  real¬ 
time. 

We  also  discovered  that  the  empowerment  was  more  than  just  a  management  fad.  It  requires  the  right  people 
with  the  right  skills  and  experiences,  in  the  right  culture  and  decision  making  environment,  to  thrive. 

A  risk-focused  culture,  intense  knowledge-sharing,  and  having  both  exploration  and  exploitation  structures  in 
place  (also  part  of  Group  3)  supports  the  empowered  individual,  however,  the  researchers  found  that  these  rapid 
practices  were  not  the  only  building  blocks  of  Group  3,  the  "Rapid  World"  Best  Practices.  Two  additional 
characteristics  of  decision-making  were  required  for  the  personnel  to  be  sufficiently  empowered  to  act  effectively 
in  a  rapid  environment:  forward-focus  and  an  urgent  decision  process. 
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4.4.1  Forward-Focused  Thinking 

The  researchers  saw  managers  and  engineers  closely  focused  on  future  milestones  that  would  quickly  inform 
them  when  the  team  was  veering  off  course.  One  example  of  forward-focused  thinking  observed  in  the 
interviews  was  that  progress  toward  test  and  test  plans  were  discussed  even  though  a  team  was  still  quite  early  in 
concept  formulation  discussions.  Another  example  was  that  deployment  specifics  were  being  mentioned  in 
casual  conversations  as  new  design  ideas  were  expressed.  Yet  another  example  is  "situational  awareness  of  the 
user",  i.e.,  the  extent  to  which  rapid  personnel  were  close  enough  to  the  user  group  (such  as  a  warfighter)  that 
they  knew  about  the  latest  information  from  the  war  zone  or  were  confident  that  they  could  anticipate  how  the 
warfighter  was  likely  to  react  to  a  new  idea.  Thus,  empowered  personnel  were  not  simply  empowered  to  make  a 
decision,  but,  as  they  were  making  the  decision,  they  needed  to  be  able  to  articulate  the  downstream  implications 
of  that  decision,  and  then  adjust  their  decision-making  accordingly  and  in  real-time. 


An  important  aspect  of  forward-focused  thinking  was  the  collective 
nature  of  the  thinking.  Observations  were  both  shared  with  other 
project  members  (as  part  of  best  practice  in  level  2),  and  were  also 
interpreted  among  the  team  members  for  the  appropriate  meaning 
and  application.  Thus,  empowered  personnel  were  not  necessarily  or 
exclusively  empowered  to  act  alone,  but  rather  to  act  after  a 
collective  interpretation. 

One  organization  articulated  its  forward  thinking  process  at  the 
enterprise  level,  as  shown  in  the  box.  The  leader  mapped  out  a 
process  that  showed  the  path  a  technology  took  from  design  to  the 
lab  to  development  to  test  to  operations  -  in  other  words,  he 
mapped  out  the  lifecycle  of  a  technology.  The  intent  was  to  always 
address  the  question  of  whether  a  technology  would  fulfill  a  current 
warfighter  need  requested  in  one  or  more  JUONS,  whether  it  could  meet  a  future  anticipated  need,  whether  it 
would  be  an  enduring  need  on  its  own,  or  whether  it  could  be  transitioned  (sooner  or  later)  into  a  program  of 
record.  This  enterprise  perspective  allowed  the  organization  to  constantly  evaluate  the  maturity  and  application 
of  the  technology  or  system,  and  to  anticipate  opportunities  to  transition.  Increasing  levels  of  SE  rigor  could  be 
included  if  the  technology  was  going  to  experience  a  longer  lifecycle,  with  the  requisite  "ilities"  of  a  traditional 
program. 


Forward-Thinking  Process 


One  Organization's  four  question  survey  for 
finished  technologies: 

1.  Who  else  can  use  it? 

2.  How  else  can  it  be  used? 

3.  What  does  this  enable  next? 

4.  How  could  this  be  used  against  us? 

The  organization  was  constantly  looking  to 
whether  the  rapid  solution  would  apply  to 
one  or  more  urgent  needs  statements,  and 
whether  it  could  transition  to  a  Program  of 
Record.  The  users  were  always  looking  for 
"what's  next?" 


4.4.2  Urgent  Decision  Process  Empowerment 

In  a  DOD  context,  often  individuals  may  not  have  the  sole  authority  to  make  a  decision;  the  decision  needs  to  be 
made  at  the  appropriate  rank.  Thus,  empowered  individuals  are  often  in  a  situation  in  which  immediate  action  is 
needed  but  they  alone  are  not  able  to  make  that  decision.  Thus,  empowered  individuals,  to  act  quickly,  must 
often  make  several  decisions:  decide  //action  is  really  needed  urgently,  decide  who  has  the  authority  to  make  the 
decision,  and  decide  how  to  get  the  decision  made  as  quickly  as  possible.  Through  forward-focused  thinking,  the 
empowered  personnel  seemed  to  be  confident  in  their  ability  to  make  the  first  decision:  decide  if  action  is  really 
needed  urgently.  The  second  decision  -  who  has  the  authority  to  make  the  decision  -  seemed  to  be  particularly 
facilitated  by  DOD  rapid  program  managers. 

These  experienced  managers  appeared  to  have  a  sufficiently  detailed  understanding  of  both  the  authority  to 
which  they  could  delegate  decision  making,  and  which  people  would  have  the  right  experiences  and  knowledge  to 
make  the  decision  at  the  lowest  possible  level  where  that  knowledge  was  available  and  could  be  acted  upon.  The 
leadership  also  had  confidence  that  their  people  had  the  skills,  experience,  context,  ability,  and  confidence  to 
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make  an  appropriate  decision  in  the  time  available  with  the  knowledge  available.  Often,  the  difference  between  a 
right  and  wrong  decision  may  not  be  known  immediately;  everyone  understood  (per  the  "psychology  of  safety" 
described  earlier)  that  a  "wrong"  decision  would  not  be  punished  and  rather  would  be  learned  from.  The  measure 
in  a  rapid  program  is  always  time:  making  decisions  in  an  expedited  manner  to  keep  the  rapid  development 
process  going. 

Personnel  in  the  program  would  often  quickly  access  these  experienced  program  managers  to  get  rapid  advice  on 
who  has  authority  to  make  a  particular  decision.  Finally,  the  empowered  individuals  needed  to  decide  how  to 
facilitate  having  the  decision  made  when  they  personally  and  individually  could  not  make  the  decision.  The 
interviews  indicated  a  variety  of  different  methods  used  to  get  a  decision  made,  including  collocating  near  the 
highest  chain  of  command.  One  interviewee  discussed  the  importance  of  collocation  and  information  sharing 
saying,  "he  was  "61  steps  from  any  program  manager  and  123  steps  from  the  PEO's  office". 

We  also  observed  an  organization  where  empowered  decision  making  failed.  In  this  example,  a  rapid  team  was 
established,  using  many  of  the  standard  Kelly  Johnson  Skunk  Works  philosophies  that  they  were  told  would 
enable  expedited  development:  small  team,  collocation,  empowerment,  and  freedom  from  bureaucracy.  Decision 
making  was  pushed  to  lower  levels,  as  the  senior  leaders  thought  this  was  a  "benefit"  to  the  team,  would  create  a 
motivated  work  force,  and  would  facilitate  faster  and  better  decision  making.  The  opposite  happened.  The  team 
"froze"  because  this  empowered  gift  was  too  far  outside  the  known  culture  and  its  behaviors,  and  they  were 
paralyzed  with  inaction.  For  example,  the  team  members  expected  to  be  given  a  set  of  known  and  validated 
requirements  upon  which  they  could  begin  their  systems  engineering  process.  The  senior  leaders  didn't  know  if 
the  problem  was  with  the  team  members  or  the  team  leader.  An  independent  review  board  recommended 
replacing  the  team.  Ultimately  the  group  leader  remained  and  the  entire  team  was  replaced  as  the  organization 
guessed  (correctly)  that  a  different  mix  of  personalities  and  experiences  was  necessary  to  thrive  in  this 
environment.  The  new  team  consisted  of  individuals  with  a  broader  set  of  experiences,  who  appeared  to  be 
comfortable  with  managing  risk  and  uncertainty,  had  demonstrated  innovation  on  previous  projects,  and  who 
were  eager  to  be  part  of  an  innovative  new  approach.  This  example  shows  the  challenges  in  just  following  a 
"script"  for  enabling  rapid  development,  and  that  there  is  a  deeper  integration  of  people,  process,  and  product 
and  this  integration  is  within  an  overall  culture  of  rapid. 


4.5  Enablers  to  Group  3 

Aspects  of  this  Go-Fast  Culture  may  also  be  enabled  by  creative  and  effective  use  of  information  technology  (IT). 
While  not  the  focus  of  our  guiding  questions  for  the  subject  matter  experts,  some  findings  from  the  literature  and 
observations  at  a  few  organizations  suggest  new  IT  tools  may  facilitate  Group  3  characteristics.  Appendix  D 
introduces  such  techniques  as  collaborative  workspaces  and  living  team  rooms,  social  media  techniques,  and 
organizational  and  systems  engineering  assessment  tools. 

It  is  important  to  note  that  the  mere  existence  of  such  tools  does  not  guarantee  that  the  team  members  will  use 
them  effectively,  or  at  all.  A  culture  and  discipline  goes  along  with  the  tools  -  if  people  believe,  and  can 
demonstrate,  that  the  tools  make  their  lives  easier,  they  are  more  bound  to  use  them.  For  example,  one  team 
used  Sharepoint  to  post  updates  on  the  project.  However,  once  they  realized  that  the  latest  versions  resided  not 
on  the  shared  drive  but  were  hoarded  on  individual  computers,  they  stopped  going  to  Sharepoint  and  instead 
went  to  individuals  to  find  information.  This  defeated  the  purpose  of  a  collaborative,  shared  space  where 
everyone  had  access  to  the  same  information  at  the  same  time  and  could  create  knowledge  from  that  data. 

The  RT-34  research  team  practiced  its  own  collaboration  using  DropBox,  weekly  telecons,  and  emails.  We 
discovered  the  most  dramatic  progress  occurred  when  the  team  met  in  person  for  an  extended  time.  This  could 
be  due  to  human  nature  and/or  because  we  had  focused  time  without  distraction  on  the  research. 
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5  Conclusions  and  Future  Research 


5.1  Conclusions  and  Recommendations 

The  following  framework  captures  all  Observations,  Findings,  and  Recommendations,  across  all  groups. 


Inferred  Characteristics 

"Go  Fast  Cultural  Best  Practices" 


Real-Time  Management 

Intense 

Knowledge  Sharing 

Risk-Tolerant 

Culture 

Organizational 

Ambidexterity 

Direct  Observations 

"Rapid  Best  Practices" 


Flexible  Acquisition 
Practices 

Contracts,  Finance,  Hiring, 
Incentive/Reward  System 


Direct  Responses 

"Organizational  Best  Practices" 


Integrated  Approach  to  Expedited  Work 

PEOPLE 

Trust,  Motivation 
Skills,  Culture 

PROCESS 

Tailor/Scale,  Requirements, 
Risk  Management,  Tech  Debt 

PRODUCT 

Mature  Tech, 
Iterative,  Affordable 

Shift  in  energy, 
commitment,  and 
knowledge. 


Business  practices  and 
leadership  drive  the 
" go-fast "  culture. 


Common  practices, 
for  rapid  or  non-rapid 
development. 


Figure  5-1:  RT-34  Expedited  SE  Framework 

Group  1:  Direct  Reponses  (Organizational  Best  Practices) 

1.  Use  Mature  Technology  -  Focus  on  the  State  of  the  Possible 

2.  Incremental  Deployment  (Development)  is  Part  of  the  Product  Plan 

3.  Strive  for  a  Defined  Set  of  Stable  Requirements  Focused  on  Warfighter  Needs 

4.  Work  to  Exploit  Maximum  Flexibility  Allowed 

5.  Designing  out  All  Risk  Takes  Forever.. .Accept  Some  Risk 

6.  Keep  an  Eye  on  "Normalization" 

7.  Build  and  Maintain  Trust 

8.  Populate  Your  Team  with  Specific  Skills  and  Experience 

9.  Maintain  High  Levels  of  Motivation  and  Expectations 

10.  The  Government  Team  Leads  the  Way 

11.  Right-size  the  Program  -  Eliminate  or  Reduce  Major  Program  Oversight 

Summary  Observation  -  Group  1:  Rapid  requires  an  integrated  approach:  People  making  judgments, 
Processes  for  task  reductions,  and  Product  aspects  focused  on  rapid  objectives. 
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Group  2:  Direct  Observations  ("Rapid  World"  Best  Practices) 

•  Not  a  single  Rapid,  but  many  different  flexible  Rapids,  with  flexible  acquisition  practices. 

Group  3:  Inferred  Characteristics  ("Go  Fast"  Cultural  Best  Practices) 

•  Rapid  DoD  Acquisition  Requires  Real-time  Management 

•  Intense  and  efficient  knowledge-sharing  is  used  to  enable  stabilization  and  synchronization  of  information 

•  Rapid  organizations  are  characterized  by  a  risk-focused  culture 

•  Rapid  organizations  are  structured  for  ambidexterity  -  one  structure  focuses  on  exploration,  and  another 
that  focuses  on  exploitation. 

Recommendations 

Based  on  the  above  observations  and  findings,  the  following  are  the  RT-34  recommendations. 

1.  Ensure  that  projects  utilize  prioritized  organizational  and  project  best  practices 

2.  Train  program  managers,  engineers  and  contracting  officers  in  organizational  and  cultural  best  practices, 
real-time  management  approaches,  and  the  different  flexibilities  that  exist. 

3.  Encourage  programs  to  intensively  share  knowledge,  have  a  risk-focused  culture,  and  create  an  ambidextrous 
organizational  structure 

4.  Share  knowledge,  experience,  mechanisms  and  lessons  learned  across  programs  and  organizations 

5.  Quantitatively  monitor  progress  in  expediting  SE  processes,  and  use  the  measurements  to  improve  both 
schedule  acceleration  and  the  ability  to  estimate  it 

6.  Use  DOD  rapid  organizations  as  a  testbed  to  introduce  digitally-enabled  solutions 

7.  Develop  the  acquisition  workforce  using  rapid  programs  to  provide  full  lifecycle  insight  and  hands-on 
experience. 

In  addition,  we  have  added  some  observations  and  findings,  not  just  from  SME  discussions,  but  from  commercial 
best  practice  and  academic  research;  these  include  aspects  of  IT  solutions  and  systems  tools.  See  Appendix  D. 


5.2  Phase  3  Plan  to  Validate  Framework  (Considerations  for  Phase  4  Implementation) 

As  was  the  RT-34  research  original  plan,  some  form  of  implementation  of  the  framework  should  be  proposed  to 
analyze  the  framework  in  action  and  iterate  it  based  on  observations  and  results  as  applied  to  a  real  DoD 
acquisition  program.  Phase  3  proposed  to  prepare  a  plan  for  both  validating  the  framework  on  a  DoD  acquisition 
program,  or  plan  for  follow-on  research.  The  actual  execution  on  a  real  DoD  acquisition  effort  was  referred  to  as 
the  unfunded  Phase  4  effort.  Also  at  this  point  in  RT-34,  we  are  now  better  positioned  to  articulate  follow-on 
research,  based  on  the  site  visits,  data  analysis,  observations  and  findings. 

While  executing  Phase  1  and  2  of  this  RT,  the  how  and  where  an  Expedited  SE  Framework  could  best  be 
demonstrated  was  continually  discussed  by  the  team.  During  the  site  visits,  the  team  assessed  interest  on  the 
part  of  organizational  leaders  to  advocate  for  a  rapid  pilot  study. 

The  following  list  may  suggest  the  diversity  of  possible  organizations  (and  subsequently)  the  lanes  of 
acquisition  that  the  SE  Expedited  Framework  could  be  applied: 

•  CubeSat  Generational  Development  (Space  Development  &  Test  Wing;  Operationally  Responsive 
Space  office,  Kirtland  AFB,  NM) 

•  Predator  ISR  Innovations  (FIQAF/A2Q,  Creech  AFB,  NV) 

•  MicroUAVs  (Eglin  AFB,  FL  /Univ  of  Florida) 

•  U.S.  Navy  Ohio  Class  Submarine 
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•  U.S.  Army  Helicopter 

•  F-35  Avionics  upgrade  (JSF  Program  Office) 

•  AFRL  Laboratory  prototyping  /ops  demonstration  effort 

•  GPS,  next  block  or  shadow  competition  of  currest  closk. 

•  USAF/SMC  program,  incorporating  Aerospace  Corp  Lessons  Learned  from  ORS 

o  Combined  with  recommended  use  of  Aerospace  Concept  Design  Center  (CDC) 

•  AF  Next  Generation  Bomber  (Air  Force  Rapid  Capabilities  Office) 

•  AF  Reusable  Booster  System  (WPAFB,  OH) 

•  Ideas  to  be  solicited  and/or  performed  at  Army  PIF  in  Huntsville. 


5.2.1  Approaches  to  Implementation 

The  researchers  do  not  expect  that  an  existing  program  will  simply  pick-up  and  apply  the  framework  completely. 
Perhaps,  though,  parts  can  be  lifted  and  applied  judiciously.  While  the  next  "big"  program  could  be  chosen  for  a 
non-traditional,  rapid  approach,  such  as  the  new  long-range  strike  bomber  managed  from  the  Air  Force  RCO  the 
framework  could  also  be  used  in  a  mixed  (hybrid)  approach.  Figure  5-1  shows  three  such  approaches. 

Using  the  next  acquisition  program  that  is  beyond  Milestone  A  or  entering  Milestone  B,  simply  compete  a  smaller 
rapid,  more  innovative  approach  simultaneously.  See  Figure  5-1  (a).  With  recent  emphasis  in  DoDI  5000.02  and 
the  Weapon  Systems  Acquisition  Reform  Act  (WSARA)  on  competition  prototyping  (DoD,  2008;  US  Congress, 
2009),  this  approach  is  particularly  attractive.  Let  small  business  and  entrepreneurs  apply  their  techniques 
simultaneously  to  an  existing  program  of  record.  There  is  no  expectation  of  "getting  something  for  nothing",  but 
rather  there  is  an  important  tradespace  of  schedule  driven  development  with  capability/  requirements. 

A  modification  to  this  approach  is  captured  in  Figure  5-2  (b)  where  the  use  of  a  rapid  approach  is  done  while  a 
larger  program  of  record  (either  new  or  existing)  is  the  long-term  planned  transition  strategy.  The  rapid  solution 
would  be  integrated  at  a  strategic  time.  This  approach  was  observed  already  during  several  interviews,  where 
rapid  offices  were  looking  for  follow-on  partnerships  or  program  integration  opportunities. 

A  combination  of  the  two  prior  concepts  produces  approach  (c)  called  "Compete  with  Integrate".  Like  (a), 
competition  fuels  the  expedited  schedule  and  innovation,  and  like  (b)  there  could  be  one  or  more  opportunities 
for  strategic  integration.  But  this  approach  continues  to  iterate  the  rapid  solution,  as  is  best  practice.  This  allows 
either  the  original  program  of  record  to  devolve,  the  rapid  iterations  to  continue  as  needed  and  stopped  once  the 
capability  gap  is  adequately  filled. 


a.  Compete  (“fly-off")  b.  Planned  transition  c.  Compete  with  Integrate 

Figure  5-2:  Approaches  to  RT-34  Framework  Implementation 
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5.2.2  The  Challenge  of  Program  Choice 

The  Phase  4  research  could  focus  the  RT-34  "rapid  solution  engineering"  on  the  highest  payoff  approach,  rather 
than  small  incremental  change.  While  more  risky,  this  approach  has  the  potential  to  make  a  big  difference  by 
demonstration  in  a  manner  that  is  irrefutable,  while  a  less  ambitious  project's  result  will  likely  be  lost  in 
irrelevance.  A  prudent  and  conservative  approach  would  be  to  select  a  rapid  hardware  demonstration  project 
that  is  easy  to  achieve,  low-cost  and  inoffensive.  But  this  may  lead  to  the  irrelevance  mentioned  above.  If  the 
chosen  demonstration  is  in  fact  easy  to  get  approved  and  funded,  and  is  acceptable  to  the  entrenched  and 
competing  conventional  acquisition  bureaucracies,  it  necessarily  must  be  a  project  that,  once  successfully 
concluded,  will  be  easy  to  ignore.  The  response  will  be  "...sure  you  can  do  rapid  prototyping  on  a  small  project, 
but  the  project  can't  be..."  -  pick  one  -  "scaled  up"  or  "put  into  production"  or  "produce  meaningful  results  for 
the  warfighter." 

Trying  not  to  be  pessimistic,  the  researchers  point  out  the  dozens  of  rapid  projects  developed  by  Scaled 
Composites  over  the  past  thirty  years,  and  note  the  sad  fact  that  essentially  none  of  them  have  actually  seen 
production  or  field  use,  either  military  or  commercial.  Each  project  was  successfully  accomplished,  generally 
within  schedule  and  budget,  but  translation  has  proved  elusive.  The  reason  perhaps  is  that  each  was  too  radical 
and  innovative  to  be  transitioned  and  integrated  into  a  conventional  military  or  commercial  "way  of  doing  things." 
Scaled  Composites  is  now  owned  by  Northrop  Grumman  Corporation;  it  will  be  interesting  to  see  if  there  can  be  a 
more  successful  transition  or  translation  of  Scaled's  innovation  into  more  traditional  products.  Interestingly,  and 
perhaps  not  so  coincidentally,  Northrop  Grumman  also  has  a  joint  venture  partnership  with  Applied  Minds,  where 
it  deliberately  uses  the  Applied  Minds'  innovative  approach  for  early  R&D,  concept  engineering,  rapid  prototyping 
-  and  in  many  cases,  this  prototyping  results  in  actual  products  for  customers. 

The  researchers  believe  a  low-risk  project  is  needed  that  both  opponents  and  proponents  of  rapid  will  embrace. 
Crafting  such  an  agreement  will  be  the  real  challenge  of  the  effort,  not  the  execution  of  the  program.  The  RT-34 
vision  of  how  to  accomplish  this  is  to  give  each  camp  what  they  need  to  support  the  effort.  The  proponent's 
position  of  rapid  is  both  easy  to  understand  and  obvious.  If  a  significant  rapid  program  can  be  implemented,  their 
careers,  claims  and  aspirations  will  be  validated.  Bringing  along  the  opponents  (acquisition  traditionalists)  is 
where  the  struggle  will  be. 

The  opponents  must  be  convinced  by  the  following  argument.  From  their  point  of  view,  a  modest  amount  of 
money  will  be  wasted,  but  if  it  doesn't  come  out  of  a  particular  budget  they  control,  it  may  not  matter  to  them. 
The  money  must  be  "someone  else's  problem"  and  so  also  must  be  the  failure,  for  they  will  be  sure  of  the  rapid 
project's  likelihood  of  failure  -  if  not  in  execution,  or  in  demo  of  operational  use,  then  in  transition  into  a  program 
of  record.  This  certainty  of  failure  is  the  "judo"  by  which  a  rapid  program  can  win  in  the  face  of  overwhelming 
odds. 

Since  the  opponents  are  convinced  of  the  certainty  of  failure  and  the  unwillingness  to  take  on  what  they  perceive 
as  too  much  risk,  they  can  be  convinced  into  supporting  the  effort  because  they  will  believe  that  an  ignominious 
failure  will  once  and  for  all  time  discredit  the  concept  of  rapid  acquisition.  Once  discredited,  they  won't  have  to 
further  face  the  threat  from  rapid  programs  over  the  balance  of  their  careers.  This  certainty  will  be  the  undoing 
of  the  conventional  program  approach,  but  only  if  rapid  can  deliver.  If  the  rapid  program  cannot  deliver,  then  this 
may  be  a  blow  to  the  rapid  approach,  but  in  the  RT-34  view,  it  is  a  risk  worthy  of  being  taken.  The  "bet"  on 
program  selection  is  laid  out  in  illustration  Figure  5-3. 

Note  that  the  assumption  of  absolute  success  or  failure  of  one  approach  or  the  other  does  not  take  into  account 
the  "experiential  development"  that  takes  place  during  a  rapid  acquisition  program.  Rapid  programs  can  be  used 
for  the  purpose  of  giving  personnel  (especially  junior  personnel)  a  suite  of  experiences  during  an  entire  acquisition 
cycle.  This  concept  is  described  in  the  next  section. 
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Succeeds 

Rapid  Adopted  by  Other  Programs 


Fails 

Rapid  Discredited 


Succeeds 

Rapid  Not  Adopted  Due 
to  Irrelevance  Argument 


Fails 

Rapid  Discredited 


Figure  5-3:  Relevant  vs  Irrelevant  RT-34  Phase  4  Demonstration 


If  this  outcome  is  accepted  as  the  likely  outcome,  then  the  next  question,  and  arguably  the  only  significant 
question,  is  what  program  could  be  infiltrated  using  the  rapid  SE  framework? 


5.2.3  Program  Recommendation  -  GPS 

Choice  of  the  program  to  validate  is  an  acutely  important  consideration.  If  a  stand-alone  program  is  proposed, 
the  risk  is  that  there  won't  be  sufficient  Congressional  or  DoD  support  for  the  program  itself,  and  it  will  be 
stillborn  or  canceled  at  the  first  opportunity.  If  the  program  is  warfighter  critical,  the  argument  can  be  made  that 
a  rapid  demonstration  is  unethical,  since  it  would  put  troops  lives  at  risk.  This  is  a  high  barrier  to  overcome.  The 
research  therefore  needs  to  look  for  an  existing  program  where  conducting  a  rapid  demonstration  would  have 
minimal  or  no  effect  on  a  critical  DoD  function,  but  one  where  a  rapid  demonstration  would  have  a  high  impact  on 
the  perceptions  of  decision  makers  and  increase  the  odds  of  further  support.  This  seemingly  incompatible  pair  of 
requirements  presents  a  conundrum.  The  Global  Positioning  System  (GPS)  may  be  a  good  solution  -  it  is  a  ready¬ 
made,  incremental,  highly  valuable  and  publicly  visible  program  of  record  to  host  a  demonstration. 

The  GPS  program  has  the  perfect  combination  of  program  attributes  to  demonstrate  the  impact  of  low  risk 
"rapid."  It  is  incremental,  with  spacecraft  replacement  happening  on  an  episodic  basic  over  decades.  It  is  highly 
visible:  everyone  has  GPS  on  their  phones,  in  their  cars,  or  on  their  aircraft,  ground  vehicle  and  ship.  The  public 
has  all  have  seen  the  precision  of  GPS-guided  ordnance  in  action  (recall  CNN  showing  the  details  of  bombing 
enemy  targets  in  Afghanistan  and  Iraq).  If  a  rapid  alternative  spacecraft  could  be  introduced  into  the  GPS 
constellation,  and  built/operated  for  10-25%  of  the  cost  of  current  spacecraft  while  providing  80-100%  of  the 
current  capability  with  minimal  technical  debt,  this  would  be  a  winner.  If  it  doesn't  work,  there  is  no  threat  to  the 
GPS  mission. 

Another  good  reason  to  use  GPS  is  there  already  is  an  existence  proof  of  being  able  to  build  a  low  cost  spacecraft 
that  can  be  part  of  a  GPS  constellation.  This  is  the  European  Galileo  precursor  spacecraft  built  by  Surrey  Satellite 
Technology  Ltd  for  the  European  Space  Agency's  "alternative  to  GPS"  program.  SSTL  originated  in  an 
academic/research  environment  at  the  University  of  Surrey,  then  became  a  separate  company,  and  now  is  an 
independent  British  company  within  the  EADS  Astrium  NV  group.  SSTL  is  the  leading  provider  of  rapid  small 
spacecraft  and  has  a  track  record  of  delivering  successful  small  satellite  missions  for  over  25  years.  What  Surrey 
has  already  accomplished  with  Galileo  can  likely  be  replicated  quickly  and  extended  by  a  U.S  smallsat  builder, 
perhaps  with  a  smart  teaming  arrangement. 

An  existence  proof  provides  the  high  cover  needed  to  prevent  overt  skepticism  that  a  demo  would  be  a  "one  time 
wonder"  that  is  too  innovative  or  too  "radical"  to  be  used  in  operations.  At  the  same  time,  as  part  of  an 
incremental  approach,  it  would  not  challenge  existing  organizations  that  it  might  actually  be  intended  to  "go 
operational".  Once  the  capabilities  of  the  system  become  highly  visible,  existing  organizations  may  even  become 
more  enthusiastic  and  start  supporting  exploration  of  options  for  "rapid  transition  to  operations."  Just  as  early 
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design  engineering  decisions  determine  the  costs  of  expediting,  early  trade  analysis  of  a  successful  demo  system 
being  used  in  operations  becomes  itself  an  existence  proof  and  platform  for  exploring  the  possibility  of  pre¬ 
planning  a  transition. 

If  a  rapid  GPS  demonstration  is  successful,  it  could  alter  the  entire  landscape  of  DoD  procurement  and  open  up 
the  possibility  that  systems  engineering  will  evolve  naturally  out  of  its  current  state  to  become  more  innovative 
and  responsive.  The  process  will  iterate.  Career  opportunities  will  develop  to  "pull  change  up  the  chain".  The 
need  for  the  right  people,  in  the  right  place  at  the  right  time  will  require  systems  engineers  to  evolve  from  "SE 
Bookkeepers"  intent  on  checking  off  boxes  and  maintaining  accountability,  into  "SE  CFOs"  capable  of  engineering 
leadership,  flexibility,  strategic  decision  making  and  organizational  savvy. 


5.3  Proposed  Future  Research 

The  RT-34  researchers  have  identified  a  number  of  additional  research  questions  and  areas  of  proposed  research. 
The  team  recommends  that  these  research  areas  be  explored  prior  to  or  in  conjunction  with  implementation  of 
the  RT-34  framework  on  a  DoD  acquisition  program. 


5.3.1  Lanes  of  Acquisition  -  How  Does  the  Framework  Apply? 

The  major  finding  of  Group  2  was  that  there  was  not  a  single  rapid,  but  many  different  flexible  rapids.  This  was 
based  on  observations  of  multiple  lanes  of  acquisition,  depending  on  the  people,  product,  and  process  context  of 
the  rapid  development  activity.  Lanes  of  DoD  Acquisition  found  in  the  RT-34  research  may  relate  to  new 
acquisition  lanes  under  consideration  for  updated  DOD  5000.  [Kendall,  NDIA  2012.]  This  raises  new  research 
questions:  how  do  the  principles  and  attributes  from  rapid  development  apply  to  different  domains  and  different 
lanes  of  acquisition.  And  do  they  apply  equally?  Additional  research  can  be  conducted  to  examine  the  principles 
and  attributes  observed  in  each  of  the  framework  levels  and  then  correlate  them  to  the  lanes  of  acquisition. 


5.3.2  Workforce  Development:  Use  Rapid  Programs  as  Training  Experience  For  Junior  Personnel 

Experiential  development  (training)  was  a  recommendation  from  this  study  -  i.e.,  Training  in  Lean  Product 
Development,  training  in  Agile  system  and  software  engineering,  and  training/  awareness  on  flexibilities  inherent 
in  hiring  and  contracting  practices.  This  effort  would  baseline  the  current  workforce  knowledge,  skills  and  abilities 
in  rapid  principles  and  aim  to  develop  a  workforce  development  strategy.  The  training  could  take  several  forms. 

Using  Rapid  Acquisition  Programs  as  an  "Experiences  Developer" 

Personnel  (from  engineers  to  program  managers  to  contracting  officers)  who  participate  in  a  rapid  program  see  a 
full  lifecycle  of  acquisition  from  problem  statement  (e.g.,  from  the  warfighter)  to  fielding  (or  maybe  even  disposal) 
in  a  very  short  time,  say,  two  years.  In  the  same  two  years  in  a  traditional  acquisition  program,  the  personnel 
maybe  see  a  portion  of  one  milestone,  with  experiences  that  consist  of  powerpoint  charts,  budget  development 
and  justification,  bureaucratic  meetings,  and  seeking  advice  from  support  contractors.  This  contrasts  with  the 
depth  and  breadth  of  experiences  that  result  from  seeing  a  program  from  start  to  finish  in  a  short  time,  with  the 
urgency  of  the  warfighter  paramount.  Thus,  rapid  programs  can  be  used  for  the  explicit  purpose  of  developing 
experiences  (with  emphasis  on  the  plurality  of  the  word  experiences)  for  the  workforce  in  seeing  an  entire 
acquisition  cycle  and  ultimately  translating  that  knowledge  into  future  programs  and  career  development. 

This  concept  emerged  as  a  result  of  observations  from  site  visits,  anecdotes  from  socializing  this  idea,  and  the 
authors'  personal  views  noting  how  much  of  their  own  experiences  and  career  were  influenced  by  working  on 
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such  programs  and  actually  "seeing"  all  the  elements  of  an  acquisition  cycle  in  a  short  time.  Three  specific 
examples  were  seen  in  the  site  visits. 

•  Organization  A.  This  program  contains  a  myriad  of  rapid  programs  within  it  (and  has  been  doing  so  for 
many  years).  This  organization  is  a  "self-contained  rapid  ecosystem"  that  includes  everything  from  design 
to  internal  contracting.  People  inside  of  this  system  experience  a  full  lifecycle  of  acquisition,  from  the 
warfighter  expressing  a  need  to  examining  the  full  realm  of  capabilities,  to  executing  on  the  program  and 
bringing  it  to  fruition  and  getting  the  capability  into  the  hands  of  the  warfighter. 

•  Organization  B.  This  organization  realized  that  it  kept  going  to  the  same  5  people  (by  name)  when  it 
needed  something  done  quickly.  This  was  not  sustainable,  and  the  organization  needed  a  way  to  replicate 
the  "attributes"  that  these  5  people  represented.  So  it  created  rapid  component  development  programs, 
on  the  order  of  18  months,  to  cycle  people  through  as  part  of  a  career  rotation.  This  served  the  purpose 
of  gaining  experience  on  a  program  from  design  to  testing,  created  more  people  with  more  skills  that 
could  be  used  on  other  longer  and  bigger  programs,  and  also  enabled  the  organization  to  develop  more 
components  that  had  opportunities  to  be  used  in  future  applications,  especially  those  that  emerged 
rapidly.  This  approach  is  very  similar  to  rotational  "leadership  development  programs"  seen  in  industry. 

As  a  result,  they  no  longer  to  go  the  same  5  by  name,  but  rather  have  a  team  of  individuals  to  go  to  by 
attribute. 

•  Organization  C.  Organization  C  also  is  focused  on  quickly  getting  capability  into  the  hands  of  the 
warfighter  faster.  This  organization  hand-picked  SMEs  to  lead  key  elements  of  the  organization  who  had 
a  career  of  experiences  that  enabled  them  to  know  how  and  when  to  tailor  and  scale,  to  understand  risks, 
and  to  work  the  system  to  get  things  done.  They  served  as  examples  for  other  members  of  the  team,  who 
had  less  experience  or  different  experience.  This  organization  also  experimented  with  a  different 
approach  to  mission  assurance,  in  which  SMEs  were  embedded  in  the  organization  to  manage  risks  in 
parallel.  This  required  a  certain  level  of  experiences  and  trust  to  succeed. 

•  University  Analogs  for  Senior  Design  Classes.  The  SERC  has  also  seen  senior  design  projects  (such  as  those 
examined  in  RT-19  Capstone)  and  the  researchers  are  familiar  with  programs  at  universities  to  quickly 
design,  develop,  and  fly  hardware  such  as  UAVs  or  small  satellites.  Small  satellites,  for  example, 
originated  out  of  the  university  environment,  with  Stanford  leading  in  the  US  and  Surrey  leading  in  the 
UK.  Now,  small  satellite  design  courses  are  popular  throughout  the  major  aerospace  engineering  schools 
both  at  the  undergraduate  and  graduate  level.  For  example,  AFIT's  senior  design  program  creates,  tests, 
verifies,  and  even  flies  small  satellites.  While  some  traditional  programs  may  consider  small  sats  to  be 
"toys"  and  not  useful  for  big  satellite  programs,  the  reality  is  that  this  platform  gives  a  full  systems 
engineering  cycle  (and  specialty  domains)  that  can  rapidly  develop  and  test  technologies  in  a  low-cost, 
low-risk  environment  that  can  be  applied  to  future  programs.  Many  universities  are  also  using  UAVs  for 
the  same  purpose  in  senior  design  classes  -  these  are  even  easier  than  small  satellites  because  they  can 
be  "flown"  outside  very  quickly  and  not  wait  for  a  launch  vehicle.  While  these  analogs  give  a  good 
perspective  on  expedited  systems  engineering,  they  do  not  necessarily  address  the  context  of  an 
acquisition  system  and  the  dynamics  of  working  with  rapid  contracting,  running  program  management 
meetings,  etc. 

So  what  do  and  did  people  learn  from  these  examples?  They  experience  a  compressed  version  of  program 
management,  systems  engineering,  contracting,  effects  of  decision  making,  importance  of  communication,  how  to 
build  and  be  part  of  a  team,  how  to  make  mistakes  and  learn  from  them,  importance  of  listening  to  the  user 
(warfighter)  and  translating  those  needs  into  capability,  managing  a  budget,  etc.  In  all  cases,  junior  personnel 
would  be  placed  side  by  side  with  experienced  personnel.  These  experiences  are  valuable  at  any  point  in  one's 
career,  but  the  point  would  be  to  see  these  experiences  earlier,  so  that  once  an  engineer,  program  manager,  or 
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contracting  officer  goes  to  a  traditional  program,  they  have  a  perspective  of  what  it  means  to  get  things  done  and 
how  to  actually  do  that.  This  has  the  potential  of  transforming  the  way  the  government  manages  programs. 

While  an  expedited  development  process  can  provide  a  deep  educational  experience  for  those  who  are  involved 
in  it,  its  effectiveness  can  be  enhanced  by  suitable  orientation  for  the  participants  to  help  inform  them  on  what  to 
expect  and  how  to  best  take  advantage  of  the  opportunities  present  in  this  environment.  Individuals  need  this 
exposure,  and  at  the  same  time,  it  is  critical  that  teams  learn  how  to  work  together  in  an  expedited  environment. 

One  means  of  providing  this  individual  and  team  training  is  through  the  use  of  simulations  and  team  building 
exercises  and  conducting  this  in  conjunction  with  (before,  during,  after)  the  hands-on  experience.  For  example, 
the  SERC  Experience  Accelerator  (SERC  RT-16)  is  being  developed  to  provide  an  open  source  environment  to 
support  such  an  individual  and  team  building  environment.  The  Experience  Accelerator  is  being  designed  to 
facilitate  the  development  of  experiences  customized  to  particular  domains  to  allow  individuals  and  teams  to 
experience  an  entire  system  life  cycle  in  a  matter  of  hours  rather  than  years.  The  objective  of  the  simulation  is  to 
provide  a  safe,  yet  authentic  environment  in  which  participants  can  make  mistakes  without  adversely  affecting 
their  careers  or  programs.  The  simulation  provides  the  participants  with  the  ability  to  view  a  program  through 
the  entire  lifecycle,  see  the  relationships  between  elements  of  the  system,  and  the  system  developing  the  system 
and  encounter  the  challenges  faced  in  an  expedited  system  development.  One  of  the  critical  aspects  of  the 
simulation  is  to  ensure  that  the  participants  are  able  to  navigate  through  the  "gray"  zone  and  create  mental 
templates  that  can  be  applied  to  similar  future  situations.  All  of  these  are  required  to  provide  experience 
acceleration  that  can  be  used  to  prepare  individuals  and  teams  for  an  expedited  system  environment.  The 
Experience  Accelerator  could  be  a  companion  to  the  rapid  development  rotation  cycle. 

Incorporating  Framework  Elements  into  Workforce  Development  and  Training  Programs 

Other  focused  "training"  programs  can  be  developed  that  provide  insight  to  and  increase  awareness  of  the 
elements  of  the  framework.  The  training  can  start  from  Level  1,  with  basic  best  business  practices,  and  expand  to 
including  tutorials  on  flexible  contracting  methods,  exploring  all  the  contracting  tools  and  mechanisms  available 
to  contracting  officers,  but  with  which  only  a  specialized  few  have  knowledge,  experience,  and  confidence  in 
implementing.  Mentoring  and  knowledge  transfer  can  also  be  incorporated  into  the  training.  These  programs 
would  go  a  long  way  to  building  the  cultural  transformations  needed  for  Level  2. 

Hypothesis:  The  Expedited  SE  framework  is  a  function  of  rapid  programs.  However,  applying  the  framework  to 
traditional  programs  does  not  necessarily  result  in  those  programs  going  more  rapid.  The  "secret  sauce"  to  rapid 
consists  of  the  people  and  team  with  a  set  of  prioritized  attributes  that  enable  the  go  fast  culture.  The  secret 
sauce  can  be  captured  through  experiential  development  of  the  workforce  that  exposes  them  to  scenarios  in 
which  to  gain  the  attributes  necessary.  Several  organizations  (both  rapid  and  traditional)  have  developmental 
programs  that  can  be  used  as  models  for  incubating  the  workforce  with  these  attributes. 

Proposed  (Draft)  Approach 

1.  Identify  how  rapid  organizations  achieve  the  Expedited  SE  Framework  concepts;  identify/  refine  any  missing 
attributes,  especially  regarding  culture  and  team,  with  the  RT-34  SMEs.  Discuss  priorities  of  these  attributes. 

•  Identify  "hybrid"  programs  (such  ISPAN  at  Hanscom  AFB)  that  appear  to  use  the  same  attributes  as  the 
Expedited  SE  Framework,  but  are  considered  "traditional"  acquisition  programs  constrained 
predominantly  by  DOD  5000. 

•  Assess  attributes  in  the  hybrid  programs  and  compare/contrast  to  the  rapid  ones. 

2.  Identify  "incubator"  opportunities  used  by  the  RT-34  organizations  or  newly  identified  hybrid  organizations  as 
a  template  for  experiential  workforce  development.  For  example  US  SOCOM's  Research  Development  Acquisition 
Center  (SORDAC)  has  a  "ghost  program"  where  young  officer  rotate  through  a  SORDAC  project,  then  deploy  to 
serve  as  a  liaison  officer  between  ops  and  acquisition.  Similarly,  US  SOCOM  creates  Distributed  Cells  (Junior 
engineers  at  various  acquisition,  research  and  test  organizations)  that  cooperatively  acquire  rapid  solutions  under 
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SORDAC  leadership.  Another  example  was  NASA  Goddard's  rapid  instrument  program  and  focus  on  hands-on 
activities  (OJT)  early  in  junior  employee  careers. 

•  This  activity  would  document  the  objectives  of  these  programs,  metrics,  capture  scenarios,  and  numbers; 
it  would  capture  lessons  learned  and  challenges.  It  would  seek  to  discuss  the  experience  with  selectees/ 
graduates,  and  capture  their  follow-on  and  previous  assignments.  This  phase  would  determine  the  impact 
of  their  rapid  acquisition  experience  on  their  career  path  and  choice.  Integrate  these  findings 
(recommendations)  with  existing  assignment,  promotion  and  career  shaping  processes. 

3.  The  results,  together  with  the  development  of  rapid  acquisition  scenarios,  can  be  inserted  into  the  "open 
source"  RT-16  Experience  Accelerator  and  used  to  develop  a  module  for  academic  course  content.  This  task 
could  also  include  student  assessments  at  various  SERC  member  institutions. 


5.3.3  Further  Iteration  and  Validation  of  the  SE  Expedited  Framework,  vl.O 

The  framework  iterated  at  least  four  times  during  the  course  of  research  as  data  were  analyzed  and  feedback 
received.  The  research  would  benefit  from  a  "Round  2"  set  of  site  visits.  The  RT-34  site  visits  were  conducted  with 
headquarter  level  organizations  (not  specific  projects).  The  responses  from  headquarter  level  people  could  have 
been  biased  toward  the  overall  context  of  rapid  development  and  acquisition.  Gathering  detailed  data  on  specific 
projects  from  site  visits  would  further  validate  and  refine  the  current  Framework.  In  particular,  this  could  help 
delve  deeper  into  the  systems  engineering  aspects  of  the  projects  by  those  individuals  who  are  focusing  on  that 
level  of  work. 

A.  Project  Level  Case  Studies.  Many  of  the  interviews  focused  at  the  SME  and  senior  levels  of  the 
organizations.  A  follow-on  study  should  relook  these  observations,  finding  and  recommendation  by 
interviewing  project  level  management  and  engineering  staff.  Using  the  case  study  method,  the  research 
collect  specific  information  on  the  current  set  questions,  but  also  on  specific  project  data. 

B.  Check  Framework  Hierarchy.  Future  research  could  continue  to  iterate  on  the  framework  and  the 
practices  identified  within  each  level  of  the  framework.  For  example,  interviewees  could  be  asked  what 
are  the  practices  they  believe  are  the  "best  ones,"  and  if  they  followed  the  practices,  in  what  order  would 
they  implement  the  practices.  This  would  provide  data  that  would  indicate  whether  the  framework  is 
hierarchical. 

C.  Examining  Success  and  Failure.  RT-34  began  with  an  assumption  that  the  framework  would  guide  a  set  of 
critical  success  factors.  It  became  apparent  during  the  data  analysis  that  the  questions  assumed  a  level  of 
success,  but  the  questioning  did  not  specifically  ask  whether  certain  practices  absolutely  resulted  in 
success  (or  failure),  and  what  were  the  criteria  for  defining  success  or  failure.  Now  that  the  framework  is 
developed  as  a  result  of  the  interviews,  questions  can  be  asked  as  to  whether  the  framework  elements 
are  indeed  contributors  to  success  or  failure,  and  if  so,  to  what  extent.  As  a  result,  one  recommendation 
for  follow-on  research  is  a  more  structured  approach  with  specific  questions  to  assess  the  use  of  the 
framework,  as  well  as  its  contribution  to  the  success  or  failure  of  a  project  as  defined  by  the  organization. 
In  particular,  one  should  try  to  ascertain  whether  the  provided  framework  is  sufficient  to  ensure  a 
successful  project,  or  whether  it  is  merely  necessary  to  ensure  a  project  does  not  fail  outright.  These  new 
questions  should  be  used  to  re-interview  the  interviewees  at  a  program  or  project  level  in  order  to  assess 
the  frame  work  in  specific  environments.  The  new  questions  can  also  be  used  to  interview  other 
programs,  including  traditional,  to  assess  how  the  framework  applies  outside  of  the  rapid  world.  The 
results  would  validate  the  framework,  or  perhaps  result  in  another  round  of  iteration  on  the  framework. 

D.  "Lanes  of  Acquisition"  Analysis  and  Tipping  Points.  Future  research  will  also  need  to  be  done  to 
determine  the  relationships  between  the  principles,  program  lanes,  expedited  capabilities  and  degrees  of 
success.  This  work  is  critical  to  determine  which  programs  are  suitable  for  expedition,  to  set  expectations 
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appropriately,  and  to  determine  which  techniques  and  tools  should  be  utilized  for  the  program.  Research 
is  needed  both  to  gather  the  necessary  information  through  surveys  and  the  mining  of  existing 
information,  performing  the  necessary  analysis  to  define  the  qualitative  and  quantitative  relationships, 
and  finally,  creating  the  tools  which  enable  practitioners  to  effectively  use  this  information. 

A  related  concept  offered  by  the  AF  Center  for  Systems  Engineering  was  a  concept  of  a  "tipping  point." 
Assuming  the  lanes  of  acquisition  are  correlated  to  rigor  of  systems  engineering,  what  program 
characteristics  "tip"  a  program  into  a  different  lane.  This  tipping  may  then  require  a  program  to  backtrack 
in  the  lifecycle  to  redo  (or  do)  various  Systems  Engineering  task. 

E.  Framework  Validation.  During  any  implementation  of  the  framework  (Section  6.2),  longitudinal  studies 
should  be  conducted  during  the  project's  lifecycle.  This  would  be  research  to  shadow  a  rapid  program, 
either  one  already  in  development,  or  one  proposed  herein.  This  would  allow  real-time  gathering  of  data 
to  refine  the  framework  and  capture  best  practices. 


5.3.4  Research  in  Integration  of  Critical  Factors  across  the  SE  Expedited  Framework 


While  there  are  many  methods  of  expediting  programs,  as  shown  in  Chapter  2,  each  supporting  one  or  several 
other  Observations,  they  are  all  dependent  on  critical  aspects  of  the  program,  the  system  characteristics  and 
environment  in  which  it  is  being  developed  and  used,  and  the  capabilities  of  the  organization  and  team 
responsible  for  the  system.  This  proposed  research  would  examine  these  relationships,  exemplified  in  the 
Equation  in  Figure  5-4,  as  a  predictive/  assessment  tool. 


Results  =  [SuccessMetrics  X  Mission/SystemType]  X 
Acquisition  Lane 


[OrgCapabilities  X  Techniques] 

V  V  J 

Expedited  Capabilities 


Figure  5-4:  Predictive/Assessment  Tool 


Without  the  appropriate  environment  in  each  of  these  areas,  many  of  the  reported  methods  for  program 
expedition  will  be  compromised  and  may  even  cause  more  delay  to  the  development  and  deployment  of  the 
system.  Therefore,  it  is  necessary  to  characterize  this  information  to  allow  those  developing  organizational  and 
program  processes  to  determine  an  appropriate  set  of  techniques  to  expedite  (or  not)  a  particular  program.  The 
following  is  a  definition  of  each  of  these  characteristics: 

Success  Metrics  -  This  defines  the  objectives  for  the  program  and  what  it  is  attempting  to  achieve.  In  some 
respects,  this  defines  the  lanes  if  you  will  where  changing  some  of  these  characteristics  could  result  in  the  need 
for  very  different  expedited  expectations  and  approaches. 

The  following  are  a  set  of  high-level  success  metrics  that  can  be  used  to  characterize  the  desired  outcome  for  a 
system: 


•  Schedule:  The  amount  of  time  required  from  commitment  to  begin  program  until  it  is  providing  the 
desired  benefits  in  the  field. 

•  Cost-Effectiveness:  systems  that  need  to  be  designed  to  be  of  minimal  cost  often  take  substantially  longer 
to  optimize  and  produce.  This  may  relate  to  the  R&D  cost  and/or  cost  of  the  system  over  its  lifecycle. 

•  Capability:  If  this  system  is  being  designed  to  replace  an  existing  system  of  similar  capability,  it  is  likely 
that  it  will  be  less  possible  to  trade  off  features  for  schedule.  Flexibility  is  a  very  important  aspect  to  allow 
for  changes  in  the  capabilities  during  the  entire  lifecycle  from  development  through  deployment.  In  any 
case,  it  is  likely  that  there  will  be  a  set  of  features  that  are  critical  to  its  success. 
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•  Scale  &  Duration  of  Deployment:  The  number  of  systems  being  deployed  and  duration  of  deployment 
impacts  the  need  for  a  robust  supply  systems  that  need  to  be  widely  deployed  over  long  periods  of  time, 
often  require  far  greater  amounts  of  planning  and  engineering  for  success. 

•  Accountability:  The  degree  to  which  costs  and  effort  need  to  be  monitored  and  accounted  for  adds  an 
increased  burden  and  adds  delay  to  the  program,  particularly  when  decisions  need  to  be  approved  at  high 
levels.  (Assurance  &  Validation) 

Mission/  System  Type  -  this  determines  to  a  large  degree  the  amount  of  work  that  needs  to  be  done.  It  could  be 
correlated  with  the  COSYSMO/  CORADMO  metrics  to  provide  some  measure  of  the  likely  duration  of  a  project.  A 
number  of  critical  system  factors  are  described  in  the  "Balancing  Agility  and  Discipline"  by  B.  Boehm  and  R.  Turner 
(2004),  which  determine  the  "home  ground"  for  agile  versus  planned  approaches,  with  the  former  often  being 
identified  as  being  the  most  expeditious.  Some  identified  factors  were: 

•  Size:  small  being  most  amenable  to  rapid  techniques 

•  Criticality:  non-critical  designs  requiring  the  least  amount  of  pre-planning 

•  Dynamism:  change  is  most  challenging  for  planful  approaches 

Organization/Personnel  Capability  -  these  define  the  capabilities  of  the  organization  and  may  limit  which 
techniques  can  be  employed  successfully.  As  noted  in  B.  Boehm  (2004),  the  capabilities  of  the  organization  are 
critical  to  the  successful  implementation  of  agile  methods.  The  same  is  true  for  expediting  programs  as  well.  The 
following  three  factors  include  the  two  factors  found  in  B.  Boehm  (2004)  and  with  the  addition  of  alignment  of 
incentives  with  program  objectives  as  noted  below: 

•  Human/tool  capabilities:  (Personnel):  it  is  noted  that  agile  approaches  require  the  continuous  presence 
of  a  critical  mass  of  scarce  experts;  this  is  especially  true  for  expedited  programs.  It  is  believed  in 
software  development,  that  a  topnotch  programmer  can  do  more  than  many  mediocre  ones  and  with 
much  higher  quality  and  far  less  supervision. 

•  Trust  (Culture):  agile  development  thrives  in  a  culture  "where  people  feel  comfortable  and  empowered 
by  having  many  degrees  of  freedom".  The  scope  of  trust  is  an  important  element  for  expeditious  behavior 
and  extends  throughout  the  organizations  in  acquisition,  development  and  deployment,  and  particularly 
in  the  interfaces  between  these  three.  As  noted  by  P.  Lencioni  ("Five  Dysfunctions  of  a  Team")  and 
others,  T rust  is  the  critical  foundation  of  teamwork  without  which  it  is  not  possible  to  effectively  team  or 
collaborate. 

•  Alignment  of  Incentives  with  Program  Objectives  (Governance):  Quite  often  the  existing  incentives  are 
to  prolong  a  program,  rather  than  complete  it  as  efficiently  and  effectively.  Alignment  must  be  in  place  at 
all  levels  to  ensure  that  all  of  the  capabilities  are  deployed  effectively  to  achieve  the  desired  outcomes. 


5.3.5  Research  Balancing  Capability,  Schedule,  Flexibility,  and  Technical  Debt 

The  ultimate  goal  of  today's  systems  and  systems  of  systems  (SoS)  is  to  provide  capabilities  to  the  stakeholders 
and  users  of  the  systems.  These  capabilities  range  from  "must-haves"  to  "nice-to-haves",  often  with 
disagreements  among  the  stakeholders  and  users  as  to  where  each  capability  lays  in  this  spectrum.  There  are 
many  choices  in  developing  and  evolving  systems  to  provide  these  desired  capabilities.  These  choices  are  typically 
related  to  development  processes  and  product  architecture  decisions.  Initial  choices  are  often  driven  by  business 
needs  such  as  time  to  market,  the  desired  level  of  performance  of  the  capabilities,  and  available  resources  such  as 
engineering  expertise  and  funding.  In  addition,  these  choices  often  result  in  longer-term  consequences  that  range 
from  good  (e.g.,  market  share  or  future  opportunities)  to  bad  (e.g.,  missed  market  share,  technical  debt,  or  a 
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failure  to  provide  the  desired  capability).  Other  times,  no  particular  attention  is  paid  to  these  choices— they 
happen  without  much  forethought,  but  still  with  the  resulting  longer-term  consequences. 

Similar  to  the  previous  research  proposal  (Section  5.3.4),  this  proposed  project  studies  the  capability  and 
affordability  tradespace  of  expediting  systems  engineering  to  reduce  schedule  and  cost,  encouraging  flexibility  in 
architecture  decisions  to  support  future  evolution  of  the  system,  and  examining  technical  debt  that  either  results 
in  later  rework  or  adversely  impacts  future  options.  In  addition,  this  research  would  examine  how  pedigreed 
systems  engineering  cost  models  can  be  used  in  the  analysis  of  this  tradespace  to  show  the  range  of  options  and 
the  resulting  consequences.  While  the  previous  proposal  examined:  1)  Success  Metrics,  2)  Mission/System  Type 
and  3)  Organizational  (expedited)  Capabilities,  this  research  would  examine  those  in  addition  to  Flexibility  and 
Technical  Debt. 

Proposed  Tasks 

This  proposed  effort  will  build  upon  three  related  SERC  research  tasks  along  with  initial  research  on  technical  debt 
conducted  by  the  Practical  Software  and  Systems  Measurement  (PSSM)  affiliates  (http://www.psmsc.com/)  to 
analyze  actual  project  data  that  has  been  previously  collected.  The  proposed  research  will  strive  to  understand 
the  limits  of  expediting  engineering  with  various  development  approaches  and  develop  new  models  based  on  the 
current  COCOMO  suite  of  cost  models  to  determine  reasonable  balances  between  expediting  engineering,  valuing 
flexibility,  and  managing  technical  debt. 

Data  analysis  could  occur  on  several  fronts: 

•  Existing  project  data  to  identify  productivity  rates  and  compare  them  the  nominal  cost  model  outputs. 

•  Cost  model  parameter  analyses  to  quantitatively  determine  to  potential  influences  of  various 
development  approaches,  techniques  and  staffing  with  respect  to  expedited  engineering,  system 
flexibility,  and  technical  debt  for  initial  system  development.  Cost  model  parameter  analyses  to  evaluate 
the  influences  of  the  various  development  approaches  and  architectures  to  system  total  ownership  costs 
over  time. 

In  addition,  more  recent  SERC  research  reports  and  other  literature  will  be  mined  for  additional  case  studies  and 
potential  influence  factors  with  respect  to  expediting  systems  engineering  and  schedule-driven  development.  The 
goal  of  these  analyses  is  to  refine  the  parameters  and  use  the  actual  project  data  and  COCOMO  suite  of  cost 
models,  illustrated  in  Table  5-1,  to  determine  effort  and  schedule  benchmarks  for  the  various  system  domains 
represented  in  the  data  sets.  In  addition,  these  analyses  will  be  used  to  develop  a  composite  model  to  investigate 
the  expediting  engineering  -  valuing  flexibility  trade  space.  This  composite  model  will  pull  from  the  COCOMO 
suite  of  cost  models  plus  the  total  ownership  cost  model  developed  for  the  SERC  Valuing  Flexibility  research  task 
(Boehm,  Lane  &  Madachy,  2011). 

The  composite  model  developed  for  this  research  effort  will  be  used  to  better  understand  the  optimal  ranges  for 
both  expediting  engineering  and  valuing  flexibility  with  respect  to  system  development  schedule,  total  ownership 
costs,  and  technical  debt.  To  do  this,  the  COCOMO  model  parameter  values  will  be  used  to  evaluate  the  SRDR, 
COCOMO,  COSYSMO,  and  student  project  data  to  understand  the  relative  level  of  expedited  engineering 
associated  with  each  project.  The  actual  project  data  will  also  be  used  to  establish  schedule  benchmarks  for  the 
various  system  domains  for  which  data  is  available.  This  information  will  be  used  to  develop  a  new  composite 
model  that  will  show  the  limits  of  expedited  engineering  with  respect  to  total  ownership  costs  and  technical  debt 
for  various  expedited  engineering  approaches. 
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Table  5-1:  COCOMO  Cost  Models 


Cost  Model 

Description 

COCOMO  II  (Boehm  et  al,  2000) 

Constructive  Cost  Model  for  software 

COQUALMO  (Boehm  et  al,  2000) 

Constructive  Quality  Model 

COPLIMO  (Boehm,  Lane  &  Koolmanojwong,  2010) 

Constructive  Product  Line  Investment  Model 

COPSEMO  (Boehm  et  al,  2000) 

Constructive  Phased  Schedule  &  Effort  Model 

CORADMO  (Boehm  et  al,  2000) 

Constructive  Rapid  Application  Development  Model 

COPROMO  (Boehm  et  al,  2000) 

Constructive  Productivity-Improvement  Model 

COCOTS  (Boehm  et  al,  2000) 

Constructive  Commercial-Off-The-Shelf  Cost  Model 

COSYSMO  (Valerdi,  2005) 

Constructive  Systems  Engineering  Cost  Model 

COSYSMO  for  SoS  (Lane,  2009) 

Constructive  System  Engineering  Cost  Model  for  SoS  Engineering 

5.3.6  Expedited  Work  Analysis 

Given  the  11  Direct  Responses,  the  research  team  then  put  these  observations  in  the  context  of  transformations 
(functions/  tasks)  needed  across  the  lifecycle.  By  examining  the  generic  development  process,  especially  for  lean 
development,  the  transformations  build  a  task  network  can  be  analyzed  for  stall,  waste,  missing  value,  lack  of 
improvement,  quality/  redo,  etc.  This  analysis  of  the  task  network  is  another  follow-on  research  proposal. 

These  are  typical  lean  development  principles.  Generally,  core  lean  activities  include: 

Specify  value:  Value  is  defined  by  customer  in  terms  of  specific  products  and  services,  but  could  include 
sustainability  or  reliability  of  the  solution,  integration  with  other  systems  (System  of  Systems),  and  for  this 
report  -  schedule.  JUONS  need  to  be  satisfied  quickly  to  reduce  risk  of  harm  or  loss  of  life. 

Value  stream  mapping:  Map  out  all  end-to-end  linked  tasks,  activities,  and  processes  necessary  for 
transforming  inputs  to  outputs  to  identify  and  eliminate  waste. 

Make  value  flow  continuously:  Having  eliminated  waste,  make  remaining  value-creating  steps  "flow".  This 
can  be  considered  having  the  lifecycle  product  iterate. 

Customers  pull  value:  Customer's  "pull"  cascades  all  the  way  back  to  the  lowest  level  supplier,  enabling  just- 
in-time  production 

Pursue  Perfection:  Pursue  continuous  process  of  improvement  striving  for  perfection 

One  engineer  called  lean  development  principles  "the  focused  application  of  common  sense".  Although  based  on 
simple  concepts,  the  complexity  arises  in  having  many  people  adopt  and  implement  the  ideas,  often  needing  to 
"retrain"  project  personnel.  Lean  thinking  is  the  dynamic,  knowledge-driven,  and  customer-focused  process 
through  which  all  people  in  a  defined  enterprise  continuously  eliminate  waste  and  create  value  (Murmun  et  al, 
2002).  More  recent  findings  (Oehmen,  2012)  have  taken  these  lean  enablers/  activities  and  added  one  additional 
-  respect  for  your  people.  With  this  background  on  lean  processes,  we  examined  removal  of  work/  waste  from 
the  lifecycle. 
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5. 3. 6.1  Project  Life-Cycle  Transformations 

The  major  objective  of  rapid  projects  is  to  reduce  the  latency  from  start  to  an  effectively  deployed  system  that 
meets  the  warfighter's  urgent  need.  The  process  of  creation  from  start  to  deployment  is  one  of  transformation; 
be  it  the  transformation  of  opportunity,  to  concept,  to  design,  to  finished  good  and  finally  to  deployed  utilization. 
Necessary  for  this  success  is  an  understanding  of  the  context  of  the  system  and  the  potential  value  creation  space. 
To  reduce  latency,  the  analysis  and  assessment  of  this  space  should  be  an  ongoing  process.  For  this  discussion, 
we  will  assume  that  this  work  occurs  outside  of  the  conceptualization  to  deployment  process. 

Once  the  system  context  and  value  creation  space  is  understood,  the  next  set  of  transformations  is  the  creation  of 
a  concept  that  provides  the  desired  value  in  the  aforementioned  context.  In  a  planned  process,  this  can  involve  a 
detailed  stakeholder  interview  and  requirements  solicitation  process,  resulting  in  a  concept  of  operations  and 
subsequent  requirements.  In  a  lightweight  process,  this  could  involve  an  intense  iterative  interaction  between 
the  stakeholders  with  the  result  being  the  creation  of  a  set  of  models,  or  other  multi-model  artifacts  that  can  be 
used  directly  in  the  subsequent  stages  of  the  process. 

The  following  set  of  transformations  is  the  architecture,  design  and  development  of  the  actual  system  (product, 
service,  etc.).  The  various  levels  of  abstraction  required  depend  on  the  scale  and  complexity  of  the  actual  system 
that  is  being  implemented.  For  heavyweight  DoD  processes,  there  are  a  number  of  documents  that  are 
contractually  required.  For  lightweight  operations,  the  documentation  might  be  the  system  itself,  and  perhaps  a 
suite  of  tests,  with  additional  embedded  comments.  The  final  transformation  is  that  of  transitioning  the  system 
into  actual  use.  This  involves  making  the  system  available  to  the  users  and  providing  the  necessary  training  and 
support.  Lastly,  closing  the  loop  is  the  evaluation  of  the  efficacy  of  the  system  which  feedbacks  into  the  context 
analysis  and  assessment  which  will  facilitate  the  creation  of  future  system's  concepts. 

A  framework  for  these  transformations  is  shown  in  Figure  5-5  (Wade  et  al,  2010). 

•  Value  -  This  includes  understanding  an  environmental  context,  understanding  factors  that  influence  the 
creation  of  value,  and  discerning  how  value  may  be  created  within  it.  To  be  successful,  the  decision  maker 
should  understand  the  total  value  proposition  which  includes  customer  needs,  not  just  customer  "wants", 
and  how  satisfying  them  brings  value  to  the  organization. 

•  Conceive  -  This  includes  the  creation  of  a  conceptual  or  abstract  design  that  creates  value.  Architects, 
stakeholders  and  particularly  user  representatives  may  be  involved  in  this  activity. 

•  Develop  -  This  includes  the  design,  manufacture  and  whatever  else  is  necessary  before  the  system  can  be 
used.  This  activity  usually  includes  engineering  and  manufacturing. 

•  Use  -  This  includes  the  distribution,  deployment,  use,  maintenance,  and  eventually  retirement  of  the 
system.  This  activity  includes  sales  and  service  and,  of  course,  the  end  user. 


Figure  5-5:  System  Life  Cycle  Activities 
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5. 3. 6.2  Life-Cycle  Latency  Reduction  Techniques 

Given  this  set  of  necessary  transformation,  how  can  the  lifecycle  process  be  expedited?  One  can  think  of  these 
transformations  in  computing  terms  in  which  many  of  the  same  latency  reduction  techniques  can  be  used.  Below 
are  six  classes  of  techniques  that  can  be  used  to  reduce  latency  as  shown  in  Table  5-2.  These  techniques  reduce 
the  total  amount  of  work  (total,  new  and  unanticipated),  increase  efficiency,  and  increase  throughput  and/or 
decrease  stall  time.  They  are  described  below. 

Table  5-2:  Program  Expedition  Techniques 


Area 

Techniques 

Impact 

Product 

1.  System  Simplification 

Reduces  total  work 

2.  Legacy  Reuse 

Reduces  total  new  work 

Processes 

3.  Transformational  Efficiency 

Increases  efficiency 

4.  Rework  Avoidance 

Reduces  rework 

People 

5.  Parallelization  &  Performance 

Increases  throughput 

6.  Improved  Decision  Making 

Reduces  "stall"  time 

These  latency  reduction  approaches  can  be  visualized  by  studying  the  systems  engineering  process  as  a  task 
network.  Figure  5-6  shows  the  network  of  system  development  tasks  with  backtracking  and  decision  gates.  While 
this  model  is  a  bit  oversimplified,  given  that  the  real  world  has  partial  dependencies  and  more  complex 
constraints,  one  can  still  use  such  a  model  to  identify  sources  of  calendar  time  reduction. 


Figure  5-6:  Systems  Engineering  Activity  Network  with  Backtracking 
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1.  Product  or  System  Simplification 

The  most  important  means  of  expediting  a  program  is  to  reduce  the  amount  of  work  that  needs  to  be 
accomplished  by  keeping  the  concept  and  design  as  simple  as  possible.  Usually,  this  involves  simplifying  how  the 
system  is  supposed  to  behave  in  the  corner  cases  of  operation.  Error  systems  tend  to  represent  a  good  deal  of 
the  complexity  of  systems,  which  is  a  great  lever  point  for  system  simplification  and  work  reduction.  This  also 
usually  requires  tradeoffs  between  requirements  and  system  complexity  as  quite  often  a  few  sets  of  requirements 
can  drive  system  complexity  and  project  schedule  well  beyond  what  they  are  individually  worth.  Success  usually 
requires  some  form  of  crisis  for  these  difficult  tradeoffs  to  be  made  as  it  is  usually  at  this  stage  of  a  program  that 
is  it  being  "sold"  to  the  many  stakeholders  and  there  is  a  great  deal  of  requirements  creep.  For  the  tradeoffs  to 
be  effective,  the  technical  assessment  team  needs  to  be  composed  of  people  who  are  experienced  in  this  type  of 
program  design,  have  worked  together  in  the  past  which  builds  trust,  are  free  to  communicate  openly  and 
honestly,  and  have  credibility  in  the  organization.  This  enables  trades  to  be  made  quickly  and  equitably  across  the 
system  space. 

2.  Legacy  Reuse 

Another  means  of  reducing  latency,  but  not  necessary  overall  work,  is  to  do  as  much  of  the  work  upfront  before 
the  project  initiation  as  possible.  This  is  broadly  applied  to  all  forms  of  legacy  including  concepts,  documents, 
designs,  components,  subsystems,  training,  etc.  This  is  where  the  concepts  of  product  lines,  modular  libraries, 
patterns,  and  standard  interfaces  come  into  play.  Depending  on  the  amount  of  intellectual  property  reuse, 
supporting  this  could  actually  require  more  effort  at  least  initially,  but  the  impact  could  be  to  greatly  reduce 
latency  from  start  to  deployment.  A  representative  product  line  approach  achieving  a  factor-of-4  decrease  in 
development  time  is  shown  in  Figure  5-7. 


Figure  5-7:  HP  Product  Line  Reuse  Investment  and  Payoff 
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In  the  late  1980s,  Hewlett  Packard  (HP)  found  that  several  of  its  market  sectors  had  product  lifetimes  of  about 
2.75  years,  while  its  waterfall  process  was  taking  4  years  for  software  development.  HP's  investment  in  a  product 
line  architecture  and  reusable  components  increased  development  time  for  the  first  two  products  in  1986-87,  but 
had  reduced  development  time  to  one  year  by  1991-92  (Lim  1998).  Similar  Platform  Based  Engineering  initiatives 
in  DoD  hardware  and  software  domains  can  have  similar  payoffs  in  time  to  first  article  development. 

Reusing  systems  engineering  assets,  model-based  systems  engineering  (MBSE)  capabilities,  and  automated  SE 
artifact  generation  have  been  shown  in  the  COSYSMO  2.0  systems  engineering  effort  estimation  model  (Fortune, 
2009)  to  eliminate  significant  sources  of  SE  effort.  For  example,  the  COSYSMO  2.0  reuse-related  percentages 
determined  by  a  combination  of  expert  judgment  and  data  analysis  are  an  added  38%  effort  to  design  SE  artifacts 
for  reuse,  and  a  savings  of  35%  for  reuse  with  modification,  57%  for  reuse  without  modification  but  a  need  for 
testing,  and  85%  for  reuse  without  modification  or  testing.  Not  all  of  this  saved  effort  will  be  on  the  critical  path, 
but  much  of  it  will  be.  However  reuse  will  require  up-front  investment  in  domain  engineering  and  structuring  SE 
artifacts  for  reuse.  A  more  detailed  discussion  of  COSYSMO  and  other  tools  is  in  Appendix  D. 

A  further  quantitative  model  for  estimating  schedule  acceleration  was  developed,  based  on  integrating  the 
schedule  predictors  in  the  previous  CORADMO  model  with  the  product,  process,  people,  and  project  factors 
identified  in  the  RT-34  research.  The  effect  of  these  factors  on  project  duration  was  calibrated  to  a  database  of  12 
agile  software  projects  from  the  AgileTek  Company.  The  projects  ranged  in  size  from  10  KSLOC  (thousands  of 
source  lines  of  code)  to  409  KSLOC,  and  had  sufficient  schedule  driver  data  to  enable  a  calibrated  set  of 
parameters  to  be  derived  that  enabled  estimation  of  eleven  of  the  12  projects  within  16%,  with  one  explainable 
outlier  off  by  53%.  The  resulting  model  was  able  to  explain  the  results  of  an  initiative  by  a  different  corporation 
that  failed  in  its  initial  attempt  to  transition  from  a  plan-driven  to  an  agile  development,  but  that  used  insights 
from  the  initial  attempt  to  achieve  a  successful  schedule  acceleration  in  its  second  attempt.  The  model, 
calibration,  and  case  study  are  presented  in  Appendix  E.  Further  research  is  proposed  to  gather  additional  data 
and  to  calibrate  the  model  to  a  broader  range  of  projects  and  organizations. 

3.  Transformational  Efficiency 

Another  means  to  reduce  the  overall  workload  is  to  reduce  the  amount  of  work  that  needs  to  be  done  in  these 
transformations.  Given  the  scale  and  complexity  of  the  system,  the  number  of  internal  abstractions  of  these 
transformational  representations  needs  to  be  minimized.  This  reduces  the  amount  of  time  that  it  takes  to 
produce  these,  but  more  importantly  reduces  the  serialization  effects  of  having  changes  ripple  through  all  of 
these.  One  could  argue  that  the  least  number  of  transformations  would  be  to  create  a  concept  that  is  embedded 
in  a  set  of  interaction  models  which  is  updated  to  include  the  actual  design,  and  then  is  used  for  deployment  and 
training.  In  this  case,  very  little  additional  intermediate  design  representations  and  documentation  may  be 
necessary. 

Business  process  re-engineering  or  lean  thinking  (Womack-Jones,  2003)  can  discover  and  eliminate  non  value¬ 
adding  tasks,  such  as  unnecessary  coordination  cycles,  purchase  approval  signatures,  or  change  control  boards 
operating  at  too  low  a  level. 

Reducing  time  per  task  can  be  addressed  through  technology  or  management.  Tools,  models,  and  automation 
can  accelerate  SE  tasks,  as  highlighted  in  the  ODDR&E  Rapid  Capability  Fielding  Tools  Study  (Carlini  et  al.,  2010). 

SE  acceleration  tools  identified  in  the  study  included  visual  modeling,  rapid  prototyping,  modeling  and  simulation, 
model-based  artifact  generation,  architecture-based  attribute  and  tradeoff  analysis,  and  integrated  SE 
environments  with  tool  interoperability.  Some  good  earlier  sources  for  reducing  time  per  task  in  the  software  area 
are  (Arthur,  1992)  and  (McConnell,  1996). 
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4.  Rework  Avoidance 


In  traditional  "Vee"  based  SE  processes,  verification  and  validation  happen  near  the  back  end  of  the  process. 
Unfortunately,  this  means  that  any  detected  issues  are  found  late  and  may  have  a  significant  impact  on  the  time 
necessary  to  deliver  a  functional  system  or  product.  The  time  necessary  to  wring  out  the  last  remaining  issues  in 
a  system  or  product  can  sometimes  is  the  dominant  factor  in  schedule  delay.  Quite  often,  expedited  programs 
use  a  continual  verification  and  validation  methodology  that  can  significantly  reduce  the  impact  and  cost  of 
detected  defects.  In  this  way,  issues  are  detected  and  resolved  as  early  as  possible  reducing  the  amount  and 
negative  impact  of  rework. 

Rework  is  perhaps  the  most  common  form  of  time-sink  that  system  development  projects  experience.  Generally, 
rework  does  not  add  value.  Thus,  a  major  challenge  and  opportunity  involves  minimizing  its  occurrence  and 
impact.  Early  and  continuous  SE  verification  and  validation  (V&V)  via  automated  analysis  tools,  modeling  and 
simulation,  prototyping,  and  independent  reviews  catch  SE  defects  earlier  when  they  are  easier  and  quicker  to  fix. 
Evidence-based  decision  milestones  provide  a  management  framework  both  to  synchronize  and  stabilize 
concurrent  SE  effort  and  to  generate  and  review  the  evidence  that  the  proposed  SE  solutions  will  enable 
satisfaction  of  the  requirements  within  the  budgets  and  schedules  of  the  plan.  Shortfalls  in  evidence  translate 
into  uncertainties  and  risks,  which  frequently  translate  into  costly  and  time-consuming  defects. 

Investments  in  process  maturity  have  been  shown  to  reduce  the  incidence  and  negative  impact  of  defects 
(Goldenson-Gibson,  2003).  The  CMMI  process  areas  of  "Validation,  Verification"  emphasize  early  defect 
identification  and  removal;  and  "Causal  Analysis  and  Resolution"  emphasizes  root  cause  analysis  to  reduce  future 
sources  of  defects.  Finally,  rework  can  be  reduced  by  tightening  convergence  loops  or  by  articulating  where 
progress  disconnects  occur.  For  example,  using  collocation  and  virtual  collaboration  technology  can  reduce  or 
surface  misunderstandings  among  SEs,  system  stakeholders,  and  independent  reviewers,  and  can  avoid  bad  fixes 
through  interactive  discussions. 

Evolutionary  definition  avoids  definition  of  details  best  left  to  downstream  increments,  such  as  the  details  of 
decision  aids  for  complex  command  and  control  systems.  Flowever,  it  is  important  to  devote  early  effort  to 
ensuring  that  the  system's  architecture  will  accommodate  anticipated  sources  of  system  change  and  growth,  thus 
avoiding  easiest-first  "mashup"  early  increments  that  foreclose  options  for  growth  and  build  up  technical  debt 
(Boehm-Bhuta,  2008). 

5.  Parallelization  and  Performance 


Another  means  of  reducing  latency  is  to  parallelize  the  work  as  much  as  is  effectively  possible  until  the  point 
where  communication  overhead  starts  to  outweigh  the  advantages  of  the  distributed  work  load.  In  this  way,  the 
latency  effects  of  serialized  efforts  can  be  minimized.  However,  over-parallelization  can  create  communication 
and  decision  making  issues;  e.g.,  the  mythical  man  month  (Brooks,  1995)  comes  to  play  here.  Also,  the 
effectiveness  of  the  resources  doing  the  transformation  can  be  increased.  This  may  be  achieved  by  using  more 
experienced  and  capable  staff  and  providing  them  with  all  the  resources  that  they  need  to  accomplish  the  job. 

On  the  management  side,  Pareto  80-20  analysis  can  be  effective  for  work  streamlining.  For  example,  if  20%  of  the 
tasks  cause  80%  of  the  time  delays,  then  task  streamlining  should  be  focused  onto  that  20%  (e.g.,  defining  and 
designing  for  critical  off-nominal  scenarios).  Particularly  for  systems  of  systems,  one  way  to  increase  effective 
parallelism  in  developing  a  number  of  components  is  to  ensure  precise,  well-validated  component  interface 
specifications  in  advance  of  detailed  component  design.  Then,  the  design  and  development  effort  for  each 
component  can  proceed  in  parallel  with  minimal  delays  due  to  interface  reconciliation  or  cross-component  ripple 
effects.  Often,  management  will  try  to  save  time  by  bringing  the  contributing-system  providers  aboard  quickly 
without  such  interface  specifications,  but  quantitative  analyses  have  shown  that  such  savings  are  eventually  much 
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more  than  wiped  out  by  late  rework  (Boehm  et  al.,  2004).  As  displayed  in  Figure  5-4,  project  activity  networks 
may  reveal  many  possible  paths  to  project  completion.  PERT/CPM  tools  and  techniques  may  help  identify  critical 
paths  in  workflow,  resource  dependency,  or  schedule.  When  activity  networks  get  too  "bushy"  (i.e.,  when  certain 
activities  have  a  high  fan-in  of  input  paths  or  fan-out  of  output  paths),  then  bottlenecks  can  occur. 

For  example,  many  early  project  review  procedures  require  mixes  of  SE  artifacts  and  numerous  other  derivative 
artifacts  such  as  plans  for  system  installation,  data  migration,  training,  cutover,  maintenance,  operations,  and 
support  to  come  together  at  a  single  formal  review.  This  frequently  requires  SEs  to  spend  much  of  their  critical 
path  time  in  tutorial  discussions  of  the  emerging  system  architecture,  when  they  need  to  focus  on  getting  the 
architecture  ready.  It  is  better  to  schedule  less-formal  SE  technical  reviews  in  advance  of  the  large,  many-artifact 
formal  reviews.  Some  early  work  on  the  derivative  artifacts  can  be  done  in  advance,  providing  useful  context  for 
the  SEs,  but  the  main  work  on  the  derivative  artifacts  can  then  be  done  once  based  on  a  well-vetted  architecture. 

Some  other  ways  to  get  time-consuming  tasks  off  the  critical  path  include  task  decomposition  and  parallelization, 
or  through  network  reconfiguration.  Examples  include  pre-positioning  of  facilities,  components,  tools,  experts,  or 
data,  which  may  add  somewhat  to  the  cost  but  be  worth  it  in  schedule  savings.  A  good  example  is 
"overinvestment"  in  reusable  components  as  discussed  in  Eliminating  Tasks. 

System  development  projects  are  sometimes  prone  to  single  point  failures,  which  can  negatively  impact 
completion  schedules.  For  example,  SE  support  environments  can  go  down  or  fail  at  untimely  moments  (a.k.a., 
"Murphy's  Law").  Similarly,  key  project  personnel  such  as  lead  system  architects  may  leave  the  company  or  be 
pulled  off  to  save  another  project.  The  basic  way  to  reduce  the  risk  of  single  point  failures  is  to  reduce  both  the 
probability  of  failure  (e.g.,  by  providing  bonuses  for  staying  with  the  project  through  completion)  and  the  impact 
of  failure  (e.g.,  by  spreading  knowledge  via  task-sharing  and  peer  reviews).  The  amount  of  risk  exposure 
(Probability  of  failure  times  Impact  of  failure)  is  a  good  way  to  prioritize  failure  risk-avoidance  efforts  such  as 
reducing  the  effects  of  personnel  turnover. 

Clearly,  every  project  would  like  its  SE  team  and  its  Integrated  Product  Teams  (IPTs)  of  stakeholders  to  consist  of 
the  best  people  possible.  Often,  though,  "best"  is  interpreted  as  "technically  strongest,"  which  often  leads  to 
continuing  arguments  among  polarized  experts  and  slow  progress.  An  excellent  set  of  criteria  for  "best"  SE  and 
IPT  team  members  is  provided  in  Air  Force  Instruction  63-123,  Evolutionary  Acquisition  for  C2  Systems  (AirForce, 
2000): 

•  empowered  -  have  the  authority  to  negotiate  for  the  organizations  that  they  represent; 

•  committed  -  provide  continuous  representation  of  their  constituencies  and  ensure  performance  of 
actions  necessary  to  achieving  group  objectives; 

•  representative  -  represent  their  entire  constituency,  not  just  a  part  or  their  personal  positions; 

•  knowledgeable  -  be  sufficiently  aware  of  group  objectives  and  have  organizational,  technical,  and 
management  expertise  to  ensure  informed  and  effective  collaboration;  and 

•  collaborative  -  operate  as  team  players  and  work  toward  win-win  solutions  for  all  stakeholders. 

As  discussed  under  Avoiding  single-point  task  failures,  a  critical  success  factor  involves  establishing  incentives  to 
attract  and  retain  the  best  SE  and  IPT  team  members.  These  can  include  bonuses  for  staying  with  the  project 
through  completion,  (e.g.  for  evolutionary  development,  one  does  not  dismiss  the  SEs  after  the  first  PDR), 
recognition  of  their  key  contributions,  and  SE  career  path  progression. 

A  true  CMMI  Level  5  SE  organization  has  accomplished  the  transition  to  a  learning  organization  via  the 
"Organizational  Innovation  and  Deployment"  process  area.  Learning  organizations  can  do  more  than  optimize 
and  manage  their  processes.  They  have  instead  cultivated  a  culture  of  continuous  improvement  and  process 
redesign  as  routine  activities,  rather  than  as  uncommon  events.  They  balance  "skating  to  where  the  puck  has 
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been"  process  improvement  via  root  cause  analysis  and  improvement  over  previous  projects,  with  "skating  to 
where  the  puck  is  going"  efforts  to  anticipate,  monitor,  and  prepare  for  future  trends. 

A  good  example  is  in  applying  trends  in  agile  methods  to  SE.  The  key  to  agility  in  complex  systems  is  for  the 
participants  to  be  able  to  operate  via  tacit  interpersonal  knowledge  and  interpersonal  trust  (Nonaka,  1991; 
Nonaka-vonKrogh,  2009),  as  compared  to  basing  collaboration  on  explicit  documented  knowledge  among 
participants  unfamiliar  with  working  with  one  another.  This  implies  both  developing  powerful  virtual 
collaboration  capabilities,  and  also  involving  the  participants  in  realistic  collaboration  exercises  that  build  up  tacit 
knowledge,  mutual  understanding,  and  trust.  Other  good  approach  for  transforming  SE  into  a  learning 
organization  is  contained  in  (Wade,  2010). 

6.  Improved  Decision  Making 

One  can  think  of  these  transformations  in  computing  terms.  Once  you  have  reduced  the  amount  of  work,  "pre¬ 
fetched"  or  pre-calculated  as  many  results  as  possible,  and  have  redistributed  the  computational  load  as  much  as 
possible  (until  the  communication  overhead  starts  to  dominate);  then  all  that  remains  to  maximally  reduce 
latency  is  to  keep  decision  making  off  the  critical  path  and  the  transformation  process  flowing  smoothly  through 
the  system.  Perhaps  the  most  effective  means  of  doing  this  is  to  distribute  decision  making  as  much  as  possible 
and  have  it  take  place  as  close  as  possible  to  the  point  where  the  elements  of  the  decision  making  are  known.  A 
critical  element  to  support  distributed  decision  making  is  open,  effective,  low-latency  communication.  In 
addition,  the  concepts  of  Kanban  can  be  applied  where  the  transformation  elements  are  pulled  through  the 
system  keeping  all  transformational  elements  actively  engaged  in  the  process  (which  increases  efficiency)  and  can 
reduce  latency.  One  could  think  of  this  as  a  data  flow  model  of  computing  or  system  creation. 

5. 3. 6. 3  Relation  of  Direct  Responses  to  Life-Cycle  Latency  Reduction  Techniques 

The  rapid  development  organizations  emphasize,  based  on  varying  organizational  or  governance  constraints,  or 
lanes  of  acquisition,  some  set  of  people,  process  and  product-related  practices.  These  are  the  direct  responses 
from  the  discussions  with  SMEs  as  described  in  Chapter  2. 

•  Product: 

1:  Use  Mature  Technology  &  Reduce  Complexity  -  Focus  on  State  of  the  Possible  (Reuse) 

2:  Incremental  Deployment  (Development)  is  Part  of  the  Product  Plan 

•  Process: 

3:  Strive  toward  a  defined  Set  of  Stable  Requirements  focused  on  Warfighter  Needs 
4:  Work  to  Exploit  Maximum  Flexibility  Allowed 
5:  Designing  out  All  Risk  Takes  Forever.. .Accept  Some  Risk 
6:  Keep  an  Eye  on  "Normalization" 

•  People: 

7:  Build  and  Maintain  Trust 

8:  Populate  Your  Team  with  Specific  Skills  and  Experience 
9:  Maintain  High  Levels  of  Motivation  and  Expectations 
10:  The  Government  Team  Leads  the  Way 

11:  Right-size  the  Program. ..Eliminate  or  Reduce  Major  Program  Oversight 

Each  of  these  approaches  has  been  mapped  with  the  six  different  techniques  as  shown  in  Table  5-3.  Note  that  a 
single  method  can  expedite  a  program  through  a  number  of  the  aforementioned  techniques. 
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Table  5-3:  Expedition  Methods 


Expedition  Methods 

Area  Direct  Responses  from  Site  Visits  #1  #2  #3  #4  #5  #6 


Product 

1:  Use  Mature  Technology 

X 

2:  Incremental  Deployment  (Development) 

X 

X 

Process 

3:  Defined  Set  of  Stable  Requirements  focused  on 

Warfighter  Needs 

X 

4:  Work  to  Exploit  Maximum  Flexibility  Allowed 

X 

X 

X 

5:  Designing  out  All  Risk  Takes  Forever 

X 

6:  Keep  an  Eye  on  "Normalization" 

X 

People 

7:  Build  and  Maintain  Trust 

X 

X 

8:  Populate  Your  Team  with  Specific  Skills  and  Experience 

X 

X 

9:  Maintain  High  Levels  of  Motivation  and  Expectations 

X 

X 

10:  The  Government  Team  Leads  the  Way 

X 

11:  Right-sizing  the  Product ....  Reduce  bureaucratic 
oversight 

X 

5.4  Summary 

Two  main  research  questions  emerged  that  resonate  with  the  sponsors: 

•  What  are  the  principles  and  attributes  that  are  seen  in  each  level  of  framework  and  do  they  apply  equally 
across  all  domains  and  all  lanes  of  acquisition? 

•  How  can  human  capital  development  improve  by  embedding  acquisition  personnel  in  programs  to 
develop  these  principles  and  attributes? 

Other  questions  arose  after  a  peer-review  "deep  dive"  of  the  RT-34  project.  These  questions  included: 

•  Is  there  a  depiction  of  the  framework  that  is  hierarchical  in  nature,  such  that  it  infers,  and  justifies, 
precedence  or  causality? 

•  How  is  DoD  rapid  acquisition  different  from  DoD  non-rapid  or  non-DoD  rapid  acquisition? 

•  How  can  a  traditional  acquisition  (large  ACAT  1)  embrace  appropriate  factors  of  rapid  organizations, 
effectively  becoming  a  "hybrid"  program? 

The  answers  to  these  questions  will  be  referred  to  any  follow-on  research.  The  SERC  and  the  RT-34  researchers 
will  work  with  the  current  and  potential  sponsors  to  agree  on  the  follow-on  research  and  the  validation  of  the 
framework.  This  will  also  include  identification  of  willing  and  able  sponsors  to  provide  additional  funding,  which 
may  also  influence  the  choice  of  follow-on  research  activity.  This  may  involve  conducting  the  follow-on  research 
as  proposed  above  in  parallel  to  lining  up  the  funding  for  a  rapid  development  pilot  program. 

The  ultimate  goal  is  for  the  research  and  the  framework  to  apply  beyond  the  Air  Force  to  the  broader  systems 
engineering  and  acquisition  communities,  in  DoD,  other  parts  of  government,  and  industry. 
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6  Epilogue  -  Enablers  and  Inhibitors  of  Systems  Engineering 


This  epilogue  captures  results  that  are  highly  related  to  the  SME  discussions,  observations  and  findings  from  a 
recent  workshop.  Rapid  fielding  or  expedited  systems  development  plays  a  new  role  in  developing  systems. 
Instead  of  polishing  and  perfecting  all  requirements  with  possible  delivery  delays,  these  urgent  needs  programs 
must  adapt  themselves  to  have  a  life  cycle  that  is  driven  by  a  "Faster,  Faster  and  Faster"  concept.  Increasingly, 
schedule  has  become  more  important  than  cost.  Organizations  have  to  fight  with  the  "time  to  market",  "respond 
to  competitor's  threats",  and  "urgent  needs".  On  one  hand,  project  managers  will  not  only  have  to  find  a  way  to 
make  things  work  through  the  expedited  lane,  but  also  have  to  understand  what  factors  are  the  accelerators  and 
what  are  hindrances  to  the  system  development  schedules.  On  the  other  hand,  they  have  to  ensure  that  these 
expedited  processes  will  not  lead  to  technical  debt,  especially  in  the  areas  of  future  flexibility,  degradation  of 
existing  capabilities,  or  rework.  This  section  will  elaborate  about  the  enablers  and  inhibitors  and  their  estimated 
impact  levels  of  three  types  of  the  expedited  systems  and  software  development:  a)  New  Single  System  b)  Existing 
Single  System  and  c)  System  of  Systems. 


6.1  Practical  Software  and  Systems  Measurement  (PSM)  Users'  Group  Conference 

During  the  recent  16th  Annual  Practical  Software  and  Systems  Measurement  (PSM)  Users'  Group  Conference,  we 
organized  a  workshop  called  Expediting  Systems  Engineering  within  an  SoS  to  explore  techniques  and  approaches 
to  quickly  engineer  critical  capabilities.  There  were  10  system  and  software  engineering  measurement  experts 
from  commercial,  military/defense,  and  academic  sectors.  The  workshop  facilitators  briefly  reviewed  the  goals  of 
the  workshop  and  the  critical  success  factors  already  identified  as  part  of  the  RT-34  research  effort.  The 
participants  were  then  invited  to  contribute  to  the  critical  success  factor  lists  in  the  form  of  enablers  and 
inhibitors  to  successful  expedited  systems  engineering  in  the  system  of  systems  (SoS)  environment.  The  "SoS 
environment"  included  new  single  systems  that  were  being  developed  to  be  part  of  the  SoS,  enhancements  to 
existing  systems  that  are  part  of  the  SoS,  and  engineering  at  the  SoS  level  (capability  engineering). 

The  workshop  captured  26  to  35  enablers  and  inhibitors  for  each  of  the  system  categories:  new  single  system 
development,  existing  single  system  development,  and  SoS  capability  development.  After  the  enabler  and 
inhibitor  factors  were  identified  for  each  of  the  three  categories,  each  attendee  rated  each  factor  "high", 
"medium"  or  "low"  to  indicate  the  level  of  impact  each  would  have  with  respect  to  expediting  in  the  respective 
category.  Then  each  rating  was  weighted  and  the  scores  from  each  attendee  combined  to  form  a  rank-ordered 
list  of  factors.  For  the  few  factors  where  the  weighted  scores  were  identical,  standard  deviations  from  the  mean 
were  used  to  order  the  factors.  The  following  describes,  in  more  detail,  the  identified  enabler  and  inhibitor 
factors  for  new  single  system,  existing  single  system,  and  SoS. 

Expediting  a  New  Single  System 

To  develop  a  new  system  starting  with  a  clean  sheet  of  paper  (e.g.,  a  Greenfield  development),  the  development 
team  has  considerable  freedom  in  their  choices  of  architecture,  non-developmental  items,  and  engineering 
processes.  Table  6-1  lists  in  rank  order,  the  expediting  enablers  and  inhibitors  associated  with  new  single  system 
development.  With  the  growing  use  of  agile  and  lean  engineering,  one  might  think  that  this  would  be  the  "silver 
bullet"  for  expedited  engineering.  Surprisingly  this  was  not  ranked  as  the  enabler  with  the  highest  impact.  In 
addition,  the  Greenfield  development  may  not  speed  through  "green  lights"  as  quickly  as  one  would  like.  As 
shown  in  the  second  column  in  Table  6-1,  there  are  several  factors  that  often  inhibit  progress  and  create  red  lights 
that  can  slow  it  down. 
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Table  6-1:  Top  25  Enablers  and  Inhibitors  for  Expediting  a  New  Single  System 


Rank 

Enablers 

Inhibitors 

1 

Rapid  Prototyping 

Requirements  Volatility 

2 

Target  hardware  lab  (experimentation  /  test  lab)  /  test 
like  you  fly  &  simulation 

Unprecedented  ness 

3 

Customer  /tech  requirements  flexibility 

Delayed  authority  to  proceed/start  with  fixed  milestone 

4 

Incremental  test  and  feedback 

Infeasible  schedule/staffing  profile 

5 

Incremental  Delivery  &  feedback 

Lack  of  Domain  Experience 

6 

Decision  making  authority 

Technology  Volatility 

7 

Best  people  /  personnel  capability 

High  numbers  of  external  interfaces 

8 

Agile/lean  approach 

Vague  Requirements 

9 

Less  context  switching  when  doing  multiple  projects 

Under  average  people  /  Personnel  Capability 

10 

Tools  and  automation 

Technology  Immaturity 

11 

Common  standard,  interface 

Conflicting  Stakeholders 

12 

Flexible  /  tailorable  rules 

Lack  of  decision  making  authority 

13 

Model-based  engineering 

large  number  of  subcontractors  /  stakeholders 

14 

Building  the  common  architecture/foundation 

Lack  of  development  infrastructure 

15 

Reusing  assets 

Rules  and  Regulations 

16 

Risk  Management 

Sequential  Development 

17 

Team  cohesion 

Classification  /  sensitivity 

18 

Business  process  reengineering  /  process  streamlining 

Fear  to  protest  the  contract  award  resulting  in  poor 
requirements 

19 

COTS 

Personnel  Turnover 

20 

Overnight  build 

Bad  RFP 

First,  let's  discuss  the  green  lights.  Rapid  Prototyping  is  rated  as  the  top  enabler.  Rapid  Prototyping  can  be  used  as 
project  feasibility  evidence  or  as  a  tool  to  show  progress,  to  test  a  proof-of-concept,  to  gather  feedback,  to  create 
shared  knowledge,  and  to  acquire  commitment  from  stakeholders.  Done  well,  an  early  rapid  prototype  of  high- 
risk  components  can  evolve  into  the  actual  system  component,  minimizing  both  risk  and  late  rework,  thereby 
compressing  the  development  schedule.  Having  an  integration/experimentation  lab  with  target  hardware  and 
simulators  early  in  the  development  process,  can  be  used  by  the  developers  to  converge  early  upon  a  feasible 
design  as  well  as  provide  a  mechanism  to  solicit  stakeholder  inputs  and  provide  feedback  to  the  stakeholders  that 
the  development  is  on  the  right  track.  Requirements  flexibility  refers  to  the  ability  to  adjust  or  negotiate 
requirements  if  there  is  evidence  that  the  original  requirements  are  unaffordable,  infeasible,  or  too  risky. 
Incremental  test  and  incremental  delivery  are  very  similar,  but  they  are  not  the  same.  Incremental  test  provides 
feedback  based  on  the  test  results  from  its  predefined  test  cases.  Incremental  delivery  provides  stakeholders' 
feedback  based  on  the  launched  or  delivered  product.  Incremental  delivery  of  a  single  system  is  easier  to  achieve 
compared  to  the  near-impossible  incremental  delivery  of  a  system  of  systems.  One  of  the  wastes  in  lean 
development  is  context  switching.  If  you  have  to  work  on  more  than  one  project  at  a  time,  each  time  you  switch 
the  context,  you  will  have  to  relearn  and  re-acquaintance  with  your  new  context.  This  task  switching  is  really  a  big 
waste  which  will  slow  down  your  task  and  prohibit  you  from  expediting  the  system  development. 

Inhibitors  or  hindrances  of  a  new  single  system  development  are  mostly  the  same  old  faces  you  may  see  in  the 
classic  risk  of  disasters'  list.  Requirements  creep  or  requirements  volatility  refers  to  modification,  addition,  or 
deletion  of  the  requirements  during  the  development  life  cycle.  The  later  the  volatility  is  introduced  the  longer 
the  delay  it  will  cause  to  the  system.  Unprecedentedness  challenges  system  engineers  both  in  technical  and  non¬ 
technical  perspectives.  When  a  project  is  a  one-of-a-kind  or  an  avant-garde  project,  it  is  difficult  for  all 
stakeholders  to  check  whether  the  product  is  developed  correctly  or  good  enough.  In  addition,  extra  effort  needs 
to  be  spent  on  research,  analysis  of  alternatives,  trial-and-error,  and  coming  up  with  learning  curve. 
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Delayed  authority  to  proceed  with  fixed  milestone  is  one  of  a  surprised,  but  common  inhibitor.  If  your  project  has 
started,  but  you  are  waiting  or  you  are  not  given  the  authority  to  work  on  the  project,  it  will  not  only  halt  the 
project,  but  it  will  also  cause  cascading  delays  in  all  project  activities.  And  it  can  be  worse  when  you  know  that  the 
deadline  is  approaching. 

Expediting  an  Existing  Single  System 

System  maintenance  or  enhancement  is  used  to  refer  to  the  work  of  modifying,  enhancing,  and  providing  cost- 
effective  support  to  the  existing  software.  System  enhancement  includes  error  corrections,  functional 
enhancements,  technical  renovations,  and  reengineering.  To  enhance  or  extend  from  an  existing  system,  some  of 
the  main  constraints  are  current  system  understanding  and  current  system  architecture  and  foundation.  The 
development  activities  could  be  either  a  major  or  minor  modification  of  a  product  after  delivery.  Table  6-2  lists 
the  enablers  and  inhibitors  for  expediting  an  existing  single  system. 

Table  6-2:  Top  25  Enablers  and  Inhibitors  for  Expediting  an  Existing  Single  System 


Rank 

Enablers 

Inhibitors 

1 

Target  hardware  lab  (experimentation  /  test  lab)  /  test  like 
you  fly  &  simulation 

Requirements  Volatility 

2 

Incremental  test  and  feedback 

High  numbers  of  external  interfaces 

3 

Incremental  Delivery  &  feedback 

Unprecedented  ness 

4 

Flexible  /  tailorable  rules 

Vague  Requirements 

5 

Agile/lean  approach 

Embedded  poor  quality  software 

6 

Rapid  Prototyping 

Conflicting  Stakeholders 

7 

Common  standard,  interface 

Delayed  authority  to  proceed/start  with  fixed 
milestone 

8 

Customer  /tech  requirements  flexibility 

Infeasible  schedule/staffing  profile 

9 

Domain  knowledge 

Technical  debt 

10 

Understanding  of  the  existing  system  and  interfaces 

Interoperability/  compatibility 

11 

Best  people  /  personnel  capability 

large  number  of  subcontractors  /  stakeholders 

12 

Less  context  switching  when  doing  multiple  projects 

Back  propagation 

13 

Tools  and  automation 

Lack  of  understanding  of  the  existing  system  and 

interfaces 

14 

Reusing  assets 

Under  average  people  /  Personnel  Capability 

15 

Team  cohesion 

Lack  of  Domain  Experience 

16 

Feasibility  Evidence,  Milestone  review 

Technology  Volatility 

17 

Model-based  engineering 

Technology  Immaturity 

18 

Colocation  of  hw  &  sw  engineers 

Classification  /  sensitivity 

19 

Development  process  tailoring/adjustment 

Multiple  operational  sites  with  different  configuration 
/  platform  /  OS 

20 

Risk  Management 

Outdated  /  stovepipe  technology 

21 

Decision  making  authority 

Personnel  Turnover 

22 

COTS 

Lack  of  decision  making  authority 

23 

Business  process  reengineering  /  process  streamlining 

Architecture  constraint  /  heritage 

24 

Outsourcing  /  surge  support 

Rules  and  Regulations 

25 

Mature  configuration  management 

Bad  documentation 
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Common  standard  or  interface  is  one  of  the  important  enabler  for  system  maintenance.  If  the  existing  system 
uses  common  interface  prototype,  it  can  save  time  in  architecting  the  extensions.  However,  if  the  existing  system 
is  using  the  APIs  or  connectors  that  are  outdated  or  uncommon,  the  development  team  needs  to  find  out  the 
possibility  of  reconnecting  the  new  system  to  the  existing  system.  Domain  knowledge  or  knowledge  about  the 
existing  system  is  also  the  key  to  expedited  systems  engineering.  On  the  opposite  side,  embedding  poor  quality  of 
system  is  one  of  the  obvious  inhibitors,  which  highly  overlaps  with  Technical  Debt  and  Back-propagation. 

Inheriting  bad  system  such  as  bad  architecture,  hardcoded  module,  spaghetti  code  requires  system  engineers  to 
payback  the  debt  or  even  worse  redo  or  reengineer  the  existing  system.  Unaware,  having  zero  knowledge  or  lack 
of  system  understanding  imposes  big  risks  on  personnel  shortfall,  architecture  complexity,  unreal  schedule,  and 
etc.  Technology  volatility  and  technology  immaturity  are  also  big  inhibitors.  Whether  it  is  a  turning  brownfield  into 
greenfield  or  continuation  of  greenfield,  technology  selection  becomes  constraints  that  the  development  team 
has  to  overcome  and  stabilize  as  soon  as  it  is  supported  by  the  feasibility  evidence. 

Expediting  a  System  of  Systems 

System  of  Systems  Engineering  (SoSE)  is  considered  to  be  a  multidisciplinary  area  and  it  includes  management  and 
organizational  aspects  as  well  as  technical  aspects  of  systems  engineering  at  the  System  of  Systems  level.  As  these 
systems  become  larger  and  larger,  the  constituent-systems  are  operationally  independent  and  managerially 
independent  and  it  requires  extra  effort  and  schedule  to  coordinate  and  synchronize  among  several  units.  Table 
6-3  shows  the  list  of  enablers  and  inhibitors  ordered  by  their  impact  level. 

Reuse  is  a  possible  shortcut  with  caveat  for  expedition.  If  those  assets  are  designed  to  be  reuse,  it  will  really  speed 
up  the  development  process.  However,  if  you  are  reusing  bad  assets,  you  will  have  to  spend  avoidable  effort  to  fix 
the  defects,  to  figure  out  the  solution,  and  eventually  you  may  end  up  with  developing  them  from  scratch.  Team 
cohesion  seems  to  affect  the  most  in  SoS  since  it  requires  high  collaboration  between  individuals  and  teams. 
Synchronization  and  Stabilization  will  give  various  SoS  teams  peace  of  mind  and  feasibility  evidence  on  their 
progress  towards  their  goals.  Common  architecture  and  foundation  is  a  good  stepping  stone  to  ensure  that  they 
have  an  established  and  strong  backbone.  It  is  impossible  for  SoS  teams  to  collect  their  information  by  using  tacit 
knowledge,  hence  Constituent  Documentation  would  be  a  great  enabler. 

Unique  inhibitors  of  SoS  such  as  high  number  of  external  interfaces,  infeasible  schedule,  large  number  of 
subcontractors,  lack  of  communication  between  teams,  poor/  unknown  heritage/pedigree,  conflicting 
stakeholders,  reflects  the  nature  of  SoS  diseconomy  of  scale. 

Table  6-3:  Enablers  and  Inhibitors  for  Expediting  a  System  of  Systems 


Rank 

Enablers 

Inhibitors 

1 

Customer  /tech  requirements  flexibility 

Lack  of  Interoperability 

2 

Rapid  Prototyping 

Lack  of  /  incompatible  standard  &  protocol 

3 

Target  hardware  lab  (experimentation  /  test  lab)  /  test 
like  you  fly  &  simulation 

Requirements  Volatility 

4 

Incremental  test  and  feedback 

Unprecedented  ness 

5 

Common  standard  and  protocol 

High  numbers  of  external  interfaces 

6 

Reusing  assets 

Infeasible  schedule/staffing  profile 

7 

Tools  and  automation 

Inability  to  test  across  systems 

8 

Common  standard,  interface 

Delayed  authority  to  proceed/start  with  fixed 
milestone 

9 

Best  people  /  Personnel  Capability 

Lack  of  Domain  Experience 

10 

Team  cohesion 

Technology  Volatility 
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11 

Synchronization  and  Stabilization 

Technology  Immaturity 

12 

flexible  /  tailorable  rules 

Vague  Requirements 

13 

Model-based  engineering 

Large  number  of  subcontractors  /  stakeholders 

14 

Building  the  common  architecture/foundation 

Lack  of  development  infrastructure 

15 

Agile/lean  approach 

Lack  of  communication  between  teams 

16 

Incremental  Delivery  &  feedback 

Classification  /  sensitivity 

17 

COTS 

Poor/  unknown  heritage/pedigree 

18 

Constituent  Documentation 

Bad  RFP 

19 

Decision  making  authority 

Personnel  Turnover 

20 

Development  process  tailoring/adjustment 

Under  average  people  /  Personnel  Capability 

21 

Risk  Management 

Conflicting  Stakeholders 

22 

Less  context  switching  when  doing  multiple  projects 

Poor  extendibility 

23 

Colocation  of  hw  &  sw  engineers  at  SOS  level 

Sequential  Development 

24 

Overnight  build 

Overspecified  requirements 

25 

Business  process  reengineering  /  process  streamlining 

Lack  of  constituent  expert 

Observation  and  Discussion 


This  section  leverages  the  list  of  enablers  and  inhibitors  by  comparing  the  similar  components  and  unique  factors 
of  the  three  systems.  Table  6-4  shows  top  10  inhibitors  of  a  new  single  system,  an  existing  single  system,  and  a 
system  of  systems.  As  highlighted  in  bold  and  italic,  these  inhibitors  are  similar  across  all  three  systems. 

Table  6-4:  Top  10  Inhibitors 


A  New  Single  System 

An  Existing  Single  System 

A  System  of  Systems 

Requirements  Volatility 

Requirements  Volatility 

Lack  of  Interoperability 

Unprecedentedness 

High  numbers  of  external  interfaces 

Lack  of  /  incompatible  standard  & 

Delayed  authority  to  proceed/start  with 

Unprecedentedness 

protocol 

Requirements  Volatility 

fixed  milestone 

Infeasible  schedule/staffing  profile 

Vague  Requirements 

Unprecedentedness 

Lack  of  Domain  Experience 

Embedded  poor  quality  software 

High  numbers  of  external  interfaces 

Technology  Volatility 

Conflicting  Stakeholders 

Infeasible  schedule/staffing  profile 

High  numbers  of  external  interfaces 

Delayed  authority  to  proceed/start 

Inability  to  test  across  systems 

Vague  Requirements 

with  fixed  milestone 

Infeasible  schedule/staffing  profile 

Delayed  authority  to  proceed/start 

Under  average  people  /  Personnel 

Technical  debt 

with  fixed  milestone 

Lack  of  Domain  Experience 

Capability 

Technology  Immaturity 

Interoperability  /  compatibility 

Technology  Volatility 

Note:  Bold-Italics  similar  across  3  categories,  Underline  reflect  unique  to  a  single  category 


For  example,  Requirements  Volatility,  Unprecendentedness,  Delayed  authority  to  proceed/start  with  fixed 
milestone,  Infeasible  schedule/staffing  profile,  and  High  numbers  of  external  interfaces,  these  factors  are 
common  inhibitors  that  can  delay  any  project.  On  the  other  hand,  each  type  of  system  has  its  own  specific 
inhibitor  as  highlighted  in  an  underlined  text.  For  a  new  single  system,  Under  Average  People  and  Technology 
Immaturity  are  the  major  hindrances  that  may  not  matter  most  in  the  other  two  systems.  For  an  existing  single 
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system,  it  is  pretty  clear  that  if  the  existing  system  has  poor  quality,  it  is  a  major  obstacle  for  the  system 
enhancement,  which  leads  to  the  next  unique  inhibitor,  technical  debt.  Inheriting  a  debt  or  substandard  system, 
the  system  maintenance  team  needs  to  spend  extra  effort  and  schedule  in  refactoring  or  restructuring  the  current 
system  and  in  turn  prevent  them  from  accelerating  their  project  development.  Lastly,  for  a  system  of  systems, 
distinctive  inhibitors  are  lack  of  interoperability  and  inability  to  test  across  systems.  These  two  inhibitors  really 
reflect  the  key  basic  concept  of  SoS.  With  so  many  systems  that  an  SoS  has  to  work  with,  if  they  don't  have  a  good 
foundation,  each  system  will  gradually  grow  apart  and  will  not  only  stop  them  from  moving  forward,  but  also 
force  them  to  step  back  to  solve  the  basic  foundation  problem.  For  SoS,  at  various  points  in  project  development, 
the  teams  need  to  stabilize  and  synchronize  their  works,  which  will  provide  a  close  loop  feedback  to  the  teams.  If 
there  is  no  one  at  check  points  or  they  are  not  able  to  check  anything  at  milestones,  it  seems  like  the  teams  are 
walking  blindly.  As  a  result,  decision  makings  will  be  delayed  or  uncommitable. 

Similarly,  there  are  also  major  overlaps  in  expediting  enablers  as  reported  in  Table  6-5.  Rapid  Prototyping,  Having 
a  target  lab,  increment  test  and  feedback,  customer/  tech  requirements  flexibility  are  among  common  best 
practices  that  one  can  use  to  expedite  the  system  development.  Similar  to  the  inhibitors,  there  are  several 
enablers  that  contribute  more  in  a  particular  system,  but  not  at  the  others.  An  SoS  requires  extremely  high 
collaboration  between  teams,  hence  it  really  needs  a  good  team  cohesion.  For  an  existing  single  system,  to  know 
and  understand  the  system  that  you  have  to  enhance  is  a  major  accelerator  otherwise  the  team  needs  to  study 
the  structure,  architecture,  its  operational  concepts  and  it  usually  takes  time  to  come  up  with  such  a  learning 
curve.  For  a  new  single  system,  since  there  is  no  problem  of  extending  current  system  or  extremely  high  number 
of  parties  to  deal  with,  the  team  can  really  move  fast  if  they  have  authority  to  make  decision  and  to  commit  with 
decision  and  be  accountable  to  their  responsibilities. 

Table  6-5:  Top  10  Enablers 


A  New  Single  System 

An  Existing  Single  System 

A  System  of  Systems 

Rapid  Prototyping 

Target  hardware  lab  /  test  like  you 
fly  &  simulation 

Customer  /tech  requirements  flexibility 

Target  hardware  lab  /  test  like  you  fly  & 
simulation 

Incremental  test  and  feedback 

Rapid  Prototyping 

Customer  /tech  requirements  flexibility 

Incremental  Delivery  &  feedback 

Target  hardware  lab  /  test  like  you  fly 
&  simulation 

Incremental  test  and  feedback 

Flexible  /Tailorable  rules 

Incremental  test  and  feedback 

Incremental  Delivery  &  feedback 

Agile/lean  approach 

Common  standard  and  protocol 

Decision  making  authority 

Rapid  Prototyping 

Reusing  assets 

Best  people  /  personnel  capability 

Common  standard,  interface 

Tools  and  automation 

Agile/lean  approach 

Customer  /tech  requirements 
flexibility 

Common  standard,  interface 

Less  context  switching  when  doing 
multiple  projects 

Domain  knowledge 

Best  people  /  Personnel  Capability 

Tools  and  automation 

Understanding  of  the  existing  system 

and  interfaces 

Team  cohesion 

Note:  Bold-Italics  similar  across  3  categories,  Underline  reflect  unique  to  a  single  category 


In  summary,  this  recent  workshop  supports  continued  findings  on  rapid  enablers  and  inhibitors,  whether 
developing  a  new  system  (see  Chapter  4  on  lanes  of  acquisition),  modifying  an  existing  system  or  working  in  the 
context  of  a  system  of  systems  (SoS).  Many  similar  themes  are  shown  across  the  aspects  of  people,  process  and 
product.  Again,  personnel  capability,  decision  making  authority,  mature  technologies,  understanding  and 
managing  requirements  are  top  enablers  as  seen  in  our  RT-34  Framework  Group  1. 
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6.2  Application  of  CORADMO  Systems  Engineering  Tool 

This  section  demonstrates  a  recent  application  on  the  use  of  the  Constructive  Rapid  Acquisition  and  Development 
Model  for  Systems  Engineering  (CORADMO-SE).  This  case  study  shows  the  use  of  the  CORADMO-SE  model  in 
explaining  the  differences  in  SE  schedule  acceleration  for  various  project  SE  approaches.  The  baseline  situation 
for  the  case  study  is  a  company  division  specializing  in  performing  early-SE  activities  for  diversified  company 
defense  applications,  generally  involving  teams  of  roughly  20  SEs.  The  division  has  been  traditionally  applying  a 
sequential  waterfall  or  Vee  model  in  defining  an  overall  system's  operational  concept  and  requirements,  and  then 
developing  a  system  architecture  that  satisfies  the  requirements.  Defense  needs  for  more  rapid  SE  have  led  the 
division  to  change  to  a  concurrent  agile  approach. 

The  baseline  situation  for  the  division  is  shown  in  the  yellow  (shaded)  M  boxes  in  Table  6-6.  The  usual  systems' 
product  factor  ratings  are:  are  moderately  complex;  sufficiently  diverse  to  make  reuse  infeasible;  non-subsettable 
so  that  low-priority  deferrals  are  infeasible;  only  able  to  use  models  vs.  documents  15%  of  the  time;  and  involving 
only  one  or  two  slightly  immature  (TRL  6)  technology  elements.  This  latter  is  the  only  factor  enabling  a  schedule 
reduction  (down  to  0.92  of  the  nominal).  The  other  three  factors  will  increase  the  required  schedule,  by  a  factor 
of  (1.09)*(1.09)*(1.05)  =  1.25.  Overall,  the  overall  impact  of  the  usual  system's  product  factors  on  SE  schedule  is 
an  increase  of  (1.25)*(0.92)  =  1.15. 

The  usual  systems'  three  process  factor  ratings  (highly  sequential  SE  processes;  largely  bureaucratic  internal  and 
external  project  and  business  processes;  moderate  SE  tool  coverage,  integration,  and  maturity:  CIM)  create 
another  net  schedule  stretch  out  of  (1.09)*(1.05)*(0.96)  =  1.10.  Their  usual  systems'  four  project  factors  (project 
SE  staff  size  between  10  and  30  people;  good  collaboration  support  across  several  metro-area  facilities;  moderate 
CIM  for  single-domain  MMPTs;  and  minimal  CIM  for  multi-domain  MMPTs)  account  for  a  net  schedule 
compression  of  (0.96)*(0.96)*(0.96)*(  1.04)  =  0.92. 

Their  people's  knowledge,  skills,  and  agility  (KSAs)  overall  ratings  for  general  SE,  single-domain  SE,  multiple- 
domain  SE,  and  team  compatibility  account  for  a  net  schedule  compression  of  (0.94)*(0.94)*  (1.06)*(0.94)  =  0.88. 
The  projects  are  evenly  balanced  between  risk-aversion  and  risk  acceptance,  leading  to  a  multiplier  of  1.0  and  no 
effect  on  the  schedule.  Overall,  then,  the  baseline  case  comes  very  close  to  the  nominal  schedule,  multiplying 
together  to  (1.15)*(1.10)*(0.92)*(0.88)*(1.0)  =  1.024. 

Table  6-6:  CORADMO-SE  Schedule  Drivers  and  Multipliers 


Accelerators/Ratings 

Very  Low 

Low 

Nominal 

High 

Very  High 

Extra  High 

Product  Factor:  Multipliers 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

Extremely 

complex 

Highly 

complex 

Mod.  complex 

Moderately 

simple 

Highly  simple 

Extremely  simple 

Element  Reuse 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate(50%) 

Considerate((70%) 

Extensive  (90%) 

Low-Priority  Deferrals 

Never 

Rarely 

Sometimes 

Often 

Usually 

Anytime 

Models  vs  Documents 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate  (50%) 

Considerate  (70%) 

Extensive  (90%) 

Key  Technology  Maturity 

>0  TRL  1,2  or  >1 
TRL  3 

1  TRL  3  or  >1 

TRL  4 

1  TRL  4  or  >2 

TRL  5 

1-2  TRL  5  or  >2 

TRL  6 

1-2  TRL  6 

All  >  TRL  7 

Process  Factor:  Multipliers 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent  Operational 
Concept,  Requirements, 
Architecture,  V&V 

Highly  sequential 

Mostly 

sequential 

2  artifacts 
mostly 
concurrent 

3  artifacts  mostly 
concurrent 

All  artifacts  mostly 
concurrent 

Fully  concurrent 

Process  Streamlining 

Heavily 

bureaucratic 

Largely 

bureaucratic 

Conservative 

bureaucratic 

Moderate 

streamline 

Mostly 

streamlined 

Fully  streamlined 

General  SE  tool  support  CIM 
(Coverage,  Integration, 
maturity) 

Simple  tools, 
weak  integration 

Minimal  CIM 

Some  CIM 

Moderate  CIM 

Considerable  CIM 

Extensive  CIM 

Project  Factors:  Multipliers 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of 
personnel) 

Over  300 

Over  100 

Over  30 

Over  10 

Over  3 

<3 

Collaboration  support 

Globally 

Nationally 

Regionally 

Metro-area 

Simple  campus, 

Largely 
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distributed 
weak  comm. , 
data  sharing 

distributed, 
some  sharing 

distributed, 

moderate 

sharing 

distributed, 
good  sharing 

strong  sharing 

collocated, 

Very  strong 
sharing 

Single-domain  MMPTs 
(Models,  Methods, 

Processes,  Tools) 

Simple  MMPTS, 
weak  integration 

Minimal  CIM 

Some  CIM 

Moderate  CIM 

Considerable  CIM 

Extensive  CIM 

Multi-domain  MMPTs 

Simple;  weak 
integration 

Minimal  CIM 

Some  CIM  or 

not  needed 

Moderate  CIM 

Considerable  CIM 

Extensive  CIM 

People  Factors:  Multipliers 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs 
(Knowledge,  Skills,  Agility) 

Weak  KSAs 

Some  KSAs 

Moderate  KSAs 

Good  KSAs 

Strong  KSAs 

Very  strong  KSAs 

Single-Domain  KSAs 

Weak 

Some 

Moderate 

Good 

Strong 

Very  strong 

Multi-Domain  KSAs 

Weak 

Some 

Moderate  or 
not  needed 

Good 

Strong 

Very  strong 

Team  Compatibility 

Very  difficult 
interactions 

Some  difficult 

interactions 

Basically 

cooperative 

interactions 

Largely 

cooperative 

Highly  cooperative 

Seamless 

interactions 

Risk  Acceptance  Factor: 
Multipliers 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

Highly  risk- 

averse 

Partly  risk- 

averse 

Balanced  risk 

aversion, 

acceptance 

Moderately  risk- 
accepting 

Considerably  risk- 
accepting 

Strongly  risk- 
accepting 

The  SE  division's  initial  change  to  a  concurrent  agile  process  approach  changes  some  of  the  yellow  boxes  in  the 
rating  scales  to  red  (Table  6-7).  For  example,  going  from  sequential  SE  to  doing  3  artifacts  (Operational  Concept, 
Requirements,  and  Architecture)  mostly  concurrently,  reduces  the  schedule  from  a  slowdown  factor  of  1.09  to  a 
speedup  factor  of  0.96,  or  a  net  speedup  of  0.96/1.09  =  0.88  of  the  baseline  schedule. 

However,  what  actually  happened  was  an  SE  schedule  slowdown  factor  of  about  15%.  In  trying  to  understand  the 
reasons  for  this,  the  company  did  a  CORADMO-SE  analysis  of  all  of  the  SE  schedule  influence  factors.  The  analysis 
found  that  the  transition  to  an  agile  SE  approach  was  flawed  in  several  ways.  Some  were  missed  opportunities  by 
addressing  only  some  of  the  improvable  SE  schedule  influence  factors,  but  not  others,  such  as  the  largely 
bureaucratic  internal  and  external  project  and  business  processes,  and  the  Low-rated  multi-domain  MMPTs  and 
KSAs.  Others  were  due  to  experiencing  some  frequent  pitfalls  in  transitioning  from  sequential,  heavyweight 
processes  to  agile  processes,  as  seen  in  the  other  red  boxes: 

•  Key  Technology  Maturity.  One  of  the  pitfalls  in  agile  development  is  premature  commitment  to 
attractive  but  immature  solutions  such  as  commercial-off-the-shelf  (COTS)  products.  These  later  require 
considerable  extra  work  and  delays  to  fix  or  replace.  The  change  to  a  Nominal  from  a  Very  High  rating 
caused  a  slowdown  factor  of  1.0/0.92  =  1.09 

•  General  SE  tool  support.  Using  a  mix  of  agile  SE  tools  and  their  traditional  SE  tools  made  their  SE  tools 
less  integrated,  for  a  slowdown  factor  of  1.0/0.96  =  1.04. 

•  General  SE  KSAs.  Their  SE  people  were  still  coming  up  the  learning  curve  in  their  agile-SE  KSAs,  for  a 
slowdown  factor  of  1.0/0.94  =  1.06 

•  Team  Compatibility.  Some  of  their  management  personnel  continued  to  use  traditional  approaches, 
contributing  another  slowdown  factor  of  1.0/0.94  =  1.06. 

As  a  result,  the  CORADMO-SE  estimate  of  their  net  slowdown  factor  was  (0.88)*(1.09)*(1.04)*(1.06)  *(1.06)  = 
1.13.  Thus,  the  CORADMO-SE  analysis  not  only  explained  their  SE  schedule  slowdown  factor  of  about  15%,  it  also 
provided  them  with  a  roadmap  of  further  agile  SE  improvements  they  could  make  to  begin  to  experience  agile  SE 
speedups,  along  with  estimates  of  the  impact  these  would  have  on  their  SE  schedules. 
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Table  6-7:  Initial  (Red)  and  Subsequent  (Green)  Agile  Changes  to  the  Corporate  Baseline  Ratings 


Accelerators/Ratings 

Very  Low 

Low 

Nominal 

High 

Very  High 

Extra  High 

Product  Factor:  Multipliers 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

Extremely 

complex 

Highly 

complex 

Mod.  complex 

Moderately 

simple 

Highly  simple 

Extremely 

simple 

Element  Reuse 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate 

(50%) 

Considerate 

(70%) 

Extensive 

(90%) 

Low-Priority  Deferrals 

Never 

Rarely 

Sometimes 

Often 

Usually 

Anytime 

Models  vs  Documents 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate 

(50%) 

Considerate 

(70%) 

Extensive 

(90%) 

Key  Technology  Maturity 

>0  TRL  1,2  or  >1 
TRL  3 

1  TRL  3  or  >1 

TRL  4 

1-2  TRL  5  or  >2 

TRL  6 

1-2  TRL  6 

All  >  TRL  7 

Process  Factor:  Multipliers 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent  Operational  Concept, 
Requirements,  Architecture,  V&V 

Highly  sequential 

Mostly 

sequential 

2  artifacts 
mostly 
concurrent 

All  artifacts 
mostly 
concurrent 

Fully 

concurrent 

Process  Streamlining 

Heavily 

bureaucratic 

Largely 

bureaucratic 

Conservative 

bureaucratic 

Moderate 

streamline 

Mostly 

streamlined 

Fully 

streamlined 

General  SE  tool  support  CIM 
(Coverage,  Integration,  Maturity) 

Simple  tools, 
weak  integration 

Minimal  CIM 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

Project  Factors:  Multipliers 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of  personnel) 

Over  300 

Over  100 

Over  30 

Over  10 

Over  3 

<3 

Collaboration  support 

Globally 
distributed 
weak  comm. , 
data  sharing 

Nationally 
distributed, 
some  sharing 

Regionally 

distributed, 

moderate 

sharing 

Metro- area 
distributed, 
good  sharing 

Simple  campus, 
strong  sharing 

Largely 
collocated, 
Very  strong 
sharing 

Single-domain  MMPTs  (Models, 
Methods,  Processes,  Tools) 

Simple  MMPTS, 
weak  integration 

Minimal  CIM 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

Multi-domain  MMPTs 

Simple;  weak 
integration 

Minimal  CIM 

Some  CIM  or  not 

needed 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

People  Factors:  Multipliers 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs  (Knowledge, 

Skills,  Agility) 

Weak  KSAs 

Moderate  KSAs 

Good  KSAs 

Strong  KSAs 

Very  strong 
KSAs 

Single-Domain  KSAs 

Weak 

Some 

Moderate 

Good 

Strong 

Very  strong 

Multi-Domain  KSAs 

Weak 

Some 

Moderate  or  not 

needed 

Good 

Strong 

Very  strong 

Team  Compatibility 

Very  difficult 
interactions 

Basically 

cooperative 

interactions 

Largely 

cooperative 

Highly 

cooperative 

Seamless 

interactions 

Risk  Acceptance  Factor: 

Multipliers 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

Highly  risk- 
averse 

Partly  risk- 
averse 

Balanced  risk 

aversion, 

acceptance 

Moderately 

risk-accepting 

Considerably 

risk-accepting 

Strongly  risk- 
accepting 

The  company's  agile-SE  improvement  goals  included  restoring  the  red  slowdown  factors  to  their  baseline  yellow 
rating  levels,  which  would  eliminate  an  overall  slowdown  factor  of  (1.09)*(1.04)*(1.06)  *(1.06)  =  1.29.  They  also 
included  further  initiatives  shown  by  the  green  boxes,  to: 

•  Perform  concurrent  V&V  along  with  concurrent  Operational  Concept,  Requirements,  and  Architecture 
activities,  for  a  speedup  factor  of  0.93/0.96  =  0.97 

•  Improve  their  largely  bureaucratic  internal  and  external  project  and  business  processes  to  be  at  least 
moderately  streamlined,  for  a  speedup  factor  of  0.96/1.04  =  0.92. 

If  they  could  achieve  all  of  these  goals,  they  could  achieve  a  speedup  factor  of  (1.024)*(0.97)*(0.92)/1.29  =  0.71. 
Their  first  attempt  to  do  this  did  not  achieve  the  full  impact,  but  brought  them  to  a  15%  speedup  factor  and 
subsequent  efforts  brought  them  to  a  29%  speedup  factor.  Thus,  the  CORADMO-SE  model  helped  them  achieve 
their  goals,  and  beyond  that  indicates  initiatives  that  could  speed  up  their  SE  activities  even  further. 
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Appendices 


Appendix  A:  Methodology 


A.l  Research  Design  and  Methodology 

The  main  focus  of  RT-34  is  to  create  a  framework  for  expedited  systems  engineering  for  rapid  capability  and 
urgent  needs.  The  research  team  set  out  to  discover  consistent  or  unique  attributes  of  rapid  organizations  and 
their  systems  engineering  processes.  How  do  these  organizations  field  military  capabilities  in  a  significantly 
reduced  time  than  the  time  it  takes  to  field  a  traditional  product?  What  products  do  they  develop?  What  systems 
engineering  processes  and  practices  do  they  follow?  What  are  their  teams  like?  Is  there  a  "secret  sauce"  to 
systems  engineering?  Are  they  just  breaking  all  the  real  and  perceived  rules  of  traditional  acquisition?  How  do 
they  tailor  these  rules  for  their  situation? 

The  overall  research  method  is  based  on  grounded  theory.  Grounded  theory  is  a  type  of  qualitative  research 
methodology  that  allows  theories  to  emerge  from  the  collected  data.  The  research  is  based  on  over  30  case 
studies  of  organizations  performing  rapid  development  on  U.S.  military  urgent  needs  programs,  in  which  field 
data  was  collected  and  initial  theories  developed.  The  case  studies  were  selected  to  understand  how  these 
organizations  work,  what  they  do,  and  what  critical  success  factors  may  be  involved  with  their  work. 

This  approach  is  useful  for  research  problems  that  are  complex  and  about  which  there  is  little  measurable  data 
(e.g.  organizational  and/or  sociotechnical  systems).  Further,  many  organizations  responding  to  Joint  Urgent 
Operational  Needs  Statements  (JUONS)  are  conducting  classified  work  and  therefore  public  information  about 
their  organization  and  projects  are  unavailable.  Conducting  in-person  site  visits  also  allowed  the  opportunity  to 
visit  facilities,  meet  project  personnel,  and  visually  experience  how  the  organizations  conducted  business. 

The  collection  of  data  comes  from  notes  during  discussions  with  the  leadership  of  these  "rapid"  organizations— 
essentially  subject  matter  experts  (SME)  in  the  field— to  discern  what  made  them  successful  and  discover  what 
drove  their  systems  engineering  processes. 

The  research  team  developed  questions  that  were  very  engineering  focused,  to  examine  the  Systems  Engineering 
technical  and  technical  management  processes  (e.g.,  risk,  requirements,  technical  decision  making,  modeling, 
documentation,  etc)  as  well  as  the  product  or  solution  architecture  (e.g.,  interfaces,  technical  maturity, 
complexity,  etc).  The  responses,  however,  nearly  always  pointed  back  to  the  people,  who  had  a  set  of  career 
experiences  and  a  way  of  thinking  that  enabled  them  to  make  appropriate  judgment  calls  on  what  systems 
engineering  was  needed,  and  that  this  process  was  enabled  through  the  culture  and  leadership  of  the 
organization.  The  responses  could  be  interpreted  by  the  researchers,  and  the  reader,  to  better  describe  system 
development  or  system  acquisition,  than  strictly  systems  engineering. 

The  research  follows  a  systematic,  yet  flexible  process  to  collect  data,  code  and  analyze  the  data,  make 
connections,  and  see  what  theories  can  be  generated.  This  "open  coding"  of  labels  is  an  important  part  of  the 
analysis  concerned  with  identifying,  naming  or  labeling,  categorizing,  and  describing  phenomena  found  in  the 
discussion  notes.  In  this  case,  the  theory  is  a  set  of  principles  for  successful  rapid  acquisition. 

Once  the  initial  principles  were  identified  from  the  data  analysis  of  the  discussions,  the  research  team  collected 
them  together  to  see  whether  they  made  sense  and  whether  the  principles  were  "new"  compared  to  what  is 
already  known  in  the  literature  and  by  the  researchers'  previous  experience  and  knowledge  of  practices  in  other 
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fields.  The  researchers  also  reviewed  other  SERC  Research  Tasks  to  identify  methods,  processes,  and  tools  (MPTs) 
or  other  research  findings  and  results  that  could  apply  to  rapid  development.  Many  of  the  principles  used  by 
rapid  organizations  were  validated  in  the  literature  search  to  be  practices  seen  in  other  fields.  Principles  were 
then  collated  into  observations.  Finally,  the  researchers  analyzed  what  these  practices  meant,  collected  them  into 
a  framework,  and  identified  findings  and  recommendations  for  each  group  of  the  framework. 

The  researchers  iterated  one  time  with  the  SMEs  to  ask  them  to  review  the  draft  report  and  validate  the 
framework.  We  asked  the  following  questions  and  aggregated  the  responses  received: 

•  What  sections  of  the  report  did  you  read? 

•  Does  the  report  reflect  what  you  told  the  researchers? 

•  What  level  of  the  framework  best  reflects  your  organization? 

•  How  would  you  rate  the  framework's  usefulness  for  developing  junior  personnel? 

•  How  would  rate  the  framework's  applicability  to  non-rapid  traditional  acquisition? 

•  What  are  the  top  3  of  the  11  findings  that  you  feel  are  the  most  important? 

•  Do  we  have  permission  to  list  your  organization  in  the  final  published  report?  Organizations  interviewed 
will  appear  in  the  report.  All  other  identifying  information  in  the  report  has  been  aggregated. 

•  If  so,  how  would  you  like  your  organization  listed? 

In  addition,  the  researchers  gathered  feedback  on  the  iterative  development  of  the  framework  at  the  NDIA 
Systems  Engineering  Conference  (October  2012,  San  Diego)  and  at  the  AIAA  Space  2012  Conference  (Pasadena, 
CA),  and  socializing  (informal  discussions)  with  other  members  of  the  rapid  development  community  in  both 
government  and  industry.  Several  interim  meetings  were  held  with  the  Air  Force  sponsors  to  further  gather  input 
from  them  on  the  user  community  and  possible  follow-on  research.  Finally,  a  “deep  dive"  review  was  held  with 
SERC  leadership  and  several  collaborators  to  review  the  research  methodology  and  framework  and  to  discuss 
potential  future  research  questions.  The  feedback  received  was  incorporated  into  this  final  report.  Two  journal 
articles  are  being  prepared  and  submitted,  to  the  IEEE  Systems  Journal  and  the  INCOSE  Systems  Engineering 
Journal.  Once  those  articles  are  accepted  and  published,  this  final  report  will  be  approved  for  public  release. 

The  overall  RT-34  research  approach  is  shown  in  the  Figure  A-l. 
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Figure  A-l:  RT-34  Research  Design  and  Methodology 
Discussion  of  Human  Subjects  Research 

DoD  issued  a  rule  amending  the  Defense  Federal  Acquisition  Regulation  Supplement  (DFARS)  to  address 
requirements  for  the  protection  of  human  subjects  involved  in  research  projects:  "Researcher  shall  not 
commence  performance  of  research  involving  human  subjects  that  is  covered  under  32  CFR  Part  219  or  that 
meets  exemption  criteria  under  32  CFR  219.101."  The  definition  of  Human  Subject  means  a  living  individual  about 
whom  an  investigator  (whether  professional  or  student)  conducting  research  obtains  (1)  Data  through 
intervention  or  interaction  with  the  individual,  or  (2)  Identifiable  private  information. 

This  research  sought  to  talk  with  organizations  that  practice  rapid  development.  Contacts  were  made  at  the 
leadership  level  of  these  organizations,  through  contacts  made  as  described  earlier.  In  all  cases,  an  email  was  sent 
with  an  overview  of  the  research  and  a  list  of  guiding  questions  for  a  site  visit  or  conversation.  Thus,  the  purpose 
of  the  research  was  known  up  front,  and  all  organizations  had  the  option  to  meet  with  us  or  not.  Throughout  this 
research  report,  the  words  "interview"  or  "survey"  is  used  colloquially,  for  lack  of  a  better  term.  These  terms  are 
used  to  convey  open  discussions  with,  sometimes  self-volunteered,  organizations  and  SMEs,  to  collect  data  about 
acquisition  projects,  processes  and  best  practices,  not  the  human  subjects.  These  "conversations"  evolved  in  real 
time,  and  each  conversation  was  different  depending  on  what  the  SME  wanted  to  discuss.  In  many  cases,  the  site 
visits  consisted  of  organizational  standard  briefings  (i.e.,  "this  is  how  we  do  business").  The  conversations  were 
focused  at  the  organizational  level,  not  on  individual  projects  or  personnel. 
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Identifiable  private  information  was  never  obtained  nor  solicited.  Likewise,  data  about  these  SMEs  was  never 
obtained  nor  solicited.  For  example,  some  questions  pertained  to  the  solution  or  product  characterization  of  an 
acquisition  project.  One  finding  was  to  reduce  time,  always  search  out  legacy  components  (possibly  flight- 
certified)  or  modify  existing  legacy  platforms.  Another  question  was  on  the  organization  of  work  during  the 
acquisition  project.  This  led  to  the  finding  to  organize  with  small  (10-12)  person  teams,  made  up  of  specific  skills 
appropriate  to  the  project.  As  the  observations  show,  when  matched  with  a  literature  search,  many  of  the 
findings  are  not  new. 

Further,  some  of  these  organizations  required  visit  requests  for  the  facility,  even  though  all  conversations  were 
conducted  at  the  unclassified  level.  The  act  of  requesting  and  submitting  a  visit  request  demonstrated  the 
volunteer  nature  of  the  visits  on  the  part  of  the  organization.  In  fact,  several  organizations  commented  that  they 
were  more  than  willing  to  talk  to  us,  because  this  was  one  of  the  rare  times  where  they  could  talk  about  rapid 
development  with  people  who  "understood  them." 

The  need  for  an  Institutional  Review  Board  (IRB)  to  screen  and  review  the  data  collection,  storage  and  processing 
protocols  was  deemed  unnecessary.  At  the  same  time,  however,  we  took  care  to  practice  good  data  collection 
because  we  were  aware  of  the  possibility  that  interviews  with  US  companies,  DoD  organizations  and  non-US 
companies  may  inadvertently  introduce  data  potentially  subject  to  export  control  restrictions.  Therefore,  we 
implemented  protocols  to  protect  distribution  of  raw  notes  and  to  separate  interview  notes  from  the 
organizations  and  individuals.  After  complete  review,  it  was  determined  that  no  export  control  data  was 
collected,  as  only  systems  and  marketing  level  information  was  obtained.  We  maintained  the  protection  and 
separation  as  part  of  good  practice. 


A. 2  Research  Phases 

RT-34  research  began  September  1,  2011,  with  the  predominance  of  the  data  collection  and  analysis  completed 
by  September  30,  2012,  with  time  designated  until  December  31,  2012  to  review  the  findings  and  confirm  the 
next  phase  of  research.  The  research  dollars  were  such  at  the  Stevens  and  USC  collaborators  were  funded  through 
December  31,  2012,  and  AFIT's  support  ended  September  30,  2012. 

The  RT-34  research  followed  the  following  phases,  which  were  set  at  the  beginning  of  the  research: 

•  Phase  1:  Short  planning  phase  to  identify  organizations  practicing  expedited  systems  engineering, 
including  visiting  selected  organizations  and  incorporating  input  from  the  SERC  Research  Council. 

•  Phase  2:  Analyze  current  state  of  the  art  in  expedited  systems  engineering  within  DoD  as  well  as  non-DoD 
and  commercial  sector,  and  developing  a  framework  for  scaling  SE  activities  in  response  to  different 
development  constraints,  such  as  reduced  development  time. 

•  Phase  3:  Prepare  a  plan  for  both  validating  the  framework  on  a  DoD  acquisition  program,  or  plan  for 
follow-on  research. 

•  Phase  4:  (Future,  Separate  Funding):  Execute  the  plan  from  Phase  3,  with  research  to  analyze  the 
framework  in  action  and  iterate  it  based  on  observations  and  results  as  applied  to  a  real  program. 

The  Phases  were  conducted  in  parallel,  and  the  research  evolved  as  the  site  visits  were  conducted,  additional  data 
were  collected,  and  observations  were  iterated.  Further,  the  site  visits  shifted  from  considering  DoD,  non-DoD 
and  commercial  sectors  to  a  focus  on  government  defense  organizations,  with  a  predominant  representation 
from  the  Air  Force. 
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As  seen  from  the  report  in  Chapters  2-4,  the  research  also  evolved,  based  on  the  responses  from  site  visits,  to 
consider  the  broader  context  of  the  people,  processes,  and  products  involved  in  rapid  development.  This 
required  some  additional  inferred  observations  from  the  research,  because  the  responses  did  not  always  directly 
answer  how  systems  engineering  was  expedited. 

Although  we  identified  several  methods  and  candidates  for  validating  the  framework  on  a  DoD  acquisition 
program,  we  discovered  that  the  framework  was  not  mature  enough  to  immediately  do  so.  Additional  research 
questions  arose  that,  once  addressed,  would  enable  the  validation  as  originally  intended  or  perhaps  validation  in  a 
new  way  of  one  of  the  new  findings.  The  additional  research  questions  are  addressed  in  Chapter  5  of  this  report. 


A. 3  Identification  of  Cases  for  Site  Visits  and  Interviews 

The  first  step  was  to  identify  organizations  conducting  rapid  development.  Organizations  were  selected  on  the 
basis  of  their: 

•  Identified  role  in  urgent  needs  projects  (preferably  responses  to  U.S.  military  JUONS) 

•  Known  ability  to  conduct  rapid  development 

•  Access  to  the  organization,  such  through  the  sponsor,  researchers,  or  referral  from  another  organization 

•  Willingness  to  participate  in  the  research. 

The  list  of  rapid  organizations  to  interview  began  with  organizations  recommended  by  the  SERC  Research  Council, 
the  SERC  sponsor,  the  Air  Force  sponsor,  personal  contacts  known  by  the  PI  and  co-PIs,  and  rapid  cell 
organizations  identified  in  Defense  Science  Board  (DSB)  and  General  Accounting  Office  (GAO)  reports.  Over  time, 
new  organizations  were  identified  based  on  recommendations  from  the  Pentagon  Systems  Engineering  Forum 
and  the  SMEs  themselves.  (This  "referral"  process  in  itself  validated  one  of  the  observations  of  the  circle  of  trust.) 

In  nearly  all  cases,  case  studies  were  conducted  at  the  headquarters  or  program  level,  not  at  the  project  level. 

This  allowed  insight  into  the  how  the  organization  responded  to  JUONS  or  conducted  rapid  development. 
Conducting  in-person  site  visits  also  allowed  an  opportunity  to  tour  facilities,  meet  additional  personnel,  and 
visually  experience  the  dynamics  of  the  organization.  Naturally,  further  research  may  determine  any  biases 
between  management  and  engineering  in  their  answers;  however,  because  these  project  teams  and  organizations 
were  relatively  small  and  flat,  we  hypothesize  this  is  may  not  be  a  strong  nuisance  factor. 

Each  case  was  presented  with  a  set  of  open-ended  questions  regarding  critical  success  factors  of  their 
organizations  and  examples  of  successful  rapid  development  projects.  Cases  were  provided  with  the  list  of 
questions  by  email,  and  a  follow-up  meeting  was  scheduled.  Discussions  were  conducted  in  person,  with  one  or 
two  conducted  over  the  phone  or  with  one  researcher  on  the  phone  and  another  in  person.  The  researchers  held 
one  to  two  hour  conversations  with  each  organization  (ranging  from  a  single  representative  to  a  small  group  of 
senior  leaders)  and  did  not  focus  on  specific  programs  or  projects.  In  some  cases,  written  responses  or  briefings 
were  provided  by  the  organization.  Some  individuals  requested  that  their  name  and  organization  be  kept 
confidential.  In  all  cases,  organizations  were  willing  to  respond  to  questions  and  have  a  discussion  knowing  that 
the  responses  and  data  would  be  aggregated  together  with  those  from  other  site  visits. 

Over  30  discussions  with  organizations  and  individuals  were  conducted,  mostly  from  the  government  and  military. 
(See  section  A.5  for  more  detail.)  The  notes  from  these  discussions  were  coded  and  tagged  based  on  patterns  of 
consistent  responses. 
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Initially,  the  list  of  organizations  included  government  (both  defense  and  civil)  and  commercial  entities.  As  time 
went  on,  however,  it  became  clear  that  the  government  organizations  were  different,  and  in  particular,  the 
military  ones  were  different  than  the  civilian  ones.  The  sponsor  expressed  a  preference  for  focusing  on  military 
urgent  needs  organizations,  because  this  was  the  population  of  interest.  These  organizations  were  also  more 
consistent  with  the  RT-34  Phase  3  and  Phase  4  future  research  intent  to  validate  an  expedited  SE  framework  on  a 
DoD  acquisition  program. 

The  full  list  of  site  visits  is  shown  in  Table  A-l,  which  reflects  25  locations,  some  with  discussion  with  multiple 
SMEs.  The  list  is  in  chronological  order,  showing  the  increased  focus  on  defense  and  Air  Force  organizations  as 
the  research  progressed. 

In  total,  the  25  site  visits  represented: 

•  15  government  defense  and  intelligence  community,  of  which  10  were  Air  Force 

•  4  industry  (all  4  conduct  work  for  defense,  civil,  and  commercial  customers) 

•  1  government  civil  space 

•  1  Federally  Funded  Research  and  Development  Center  (FFRDC) 

•  1  educational  institution 

•  with  the  remaining  3  "site  visits"  conducted  at  conferences  incorporating  multiple  individuals  from  all  of 
these  sectors. 

Note  that  examining  the  characteristics  of  military  organizations  -  particularly  on  how  they  are  organized  and 
conduct  business  -  is  an  area  of  potential  future  literature  search  and  research.  Such  characteristics  may  include 
the  government  acquisition  process  and  rules;  Volunteer  "army";  officers,  enlisted  personnel,  civilians, 
contractors;  and  the  government  role  as  acquirers  and  operators  not  the  builders. 


A. 4  Development  of  Questionnaire 

Lane  et  al.  2010  examined  critical  success  factors  of  five  different  organizations  developing  rapid,  innovative 
solution  and  concluded  that  successful,  high  performance  organizations  share  certain  characteristics  in  the 
development  of  innovative  systems: 

•  Driven  by  business  value 

•  Take  calculated  risks 

•  Proactive  management  and  small  agile  teams 

•  Look  for  solution  patterns  that  can  be  re-used 

•  Follow  concurrent  engineering  practices 

•  Provide  culture  and  environment  that  supports  innovation. 

These  findings  parallel  Kelly  Johnson's  famous  "14  Rules  of  Management"  developed  in  the  Lockheed  Skunk 
Works  program,  where  his  motto  was  "Be  quick,  be  quiet,  and  be  on  time."  [Johnson] 

The  questions  posed  by  Lane  et  al  [Lane  et  al  2010]  became  the  starting  point  for  the  lines  of  inquiry  to  the 
organizations  selected  as  case  studies.  The  expedited  SE  research  used  these  basic  factors  as  a  guide  and  created 
a  new  list  of  questions  to  go  into  the  next  level  of  detail  of  "why"  or  "how"  a  characteristic  may  be  important.  The 
list  of  questions  guided  open  discussions,  and  did  not  specifically  force  closed  responses  to  each  question. 

The  original  list  of  expedited  questions  is  in  Attachment  1. 
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A. 5  Refinement  of  Questions 

Within  the  first  week  of  interviews,  obvious  patterns  began  to  emerge  from  the  interviews.  These  patterns  were 
clustered  into  4  major  categories:  product,  process,  people,  and  project.  The  questions  were  iterated  based  on 
these  questions.  This  process  is  typical  of  a  pilot  test  of  questions.  The  questionnaire  shown  in  Attachment  2  was 
used  in  all  other  interviews  to  date. 

As  the  questions  are  open-ended,  each  interview  was  allowed  to  proceed  according  to  the  desire  of  the  SME  and 
the  line  of  answers  that  emerged  during  the  discussion. 


A. 6  Collection  of  Data  into  Database,  Data  Mining  &  Cluster  Analysis 

Notes  were  hand-written  by  the  researchers  during  the  site  visits.  Many  of  the  organizations  were  in  secure 
facilities,  and  thus  recording  devices  were  not  permitted.  Hand-writing  of  notes  can  introduce  human  error  and 
unintended  bias  into  the  data  collection.  For  example,  the  researcher  could  miss  an  important  word,  translate 
words  into  his  or  her  own  words,  or  hear  the  same  words  multiple  times  and  stop  writing  it  down  (thinking:  "I 
already  have  that  point"). 

An  Access  Database  was  developed  to  collect  the  site  visit  data.  The  database  tags  the  responses  according  to  the 
four  categories  (product,  process,  people,  and  project),  with  further  tags  of  key  words  that  emerged  as  patterns  in 
the  interview.  All  organizations  and  interviewees  were  coded  such  that  the  identities  were  obscured. 

Notes  from  over  30  discussions  with  government  and  industry  rapid  organizations  and  individuals  were  input  to 
the  database.  Notes  were  transcribed  word  for  word  into  the  database,  with  one  person  doing  the  transcription. 
In  cases  where  multiple  researchers  were  present  at  a  site  visit,  the  transcriber  would  compare  the  notes  and 
merge  them  together.  This  was  not  a  perfect  process,  as  the  transcriber  could  have  misinterpreted  the  notes.  To 
mitigate  this  potential  error,  one  researcher  reviewed  all  the  notes  and  the  database  for  consistency  and  to 
remove  any  duplication.  This  same  researcher  was  present  in  all  but  one  of  the  discussions,  and  thus  was  in  the 
best  position  to  make  this  judgment  call  on  the  notes. 

These  words  were  also  inserted  into  "Wordle,"  a  on-line,  free  word  count  software,  to  visualize  any  initial  patterns 
in  a  word  cloud.  The  purpose  of  using  this  qualitative  tool  was  to  give  a  visual  representation  of  prominent  words 
from  the  interviews,  allowing  initial  insight  to  key  words  and  patterns.  This  visualization  was  not  used  to 
determine  the  11  direct  response  observations  described  in  Chapter  2. 

The  following  two  figures  show  the  Wordle  diagram  at  two  points  in  time.  The  first  figure  is  in  March  2012,  with 
14,500  words  from  site  visit  notes.  This  shows  "requirements"  being  prominent. 

The  second  figure  is  from  July  2012,  when  the  full  set  of  notes  had  been  captured  in  the  database,  with  over 
23,000  words  from  the  notes.  At  this  point  in  time,  with  a  more  complete  database,  "people"  shows  up  as  the 
most  prominent  word. 
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Figure  A-2:  Word  Cloud  of  Over  30  Interviews  with  Gov't/  Industry  Rapid  Acquisition  Organizations,  with  Over 

14,500  words  from  interview  notes  (as  of  March  2012) 


Figure  A-3:  Word  Cloud  from  Over  30  Interviews  of  Gov't/  Industry  Rapid  Acquisition  Organizations,  with  Over 

23,000  words  from  interview  notes  (as  of  July  2012) 

The  results  of  the  word  cloud  are  simplistic,  as  some  words  are  repeated  in  singular  and  plural  form,  or  with 
capital  letters,  and  thus  the  true  patterns  may  be  even  stronger  than  this  represents.  For  example, 
"requirements"  appears  as  the  most  dominant  word  in  the  first  figure.  By  treating  Requirements  (capital  R), 
requirement  (singular),  and  Requirement  (capital  R  and  singular)  all  as  the  same  word,  "requirements"  may 
appear  as  an  even  stronger  pattern.  The  same  can  be  said  for  the  next  strongest  word:  "People."  There  are 
several  words  that  may  be  synonyms  for  people,  such  as  customer  contractor,  leadership,  experts,  etc. 

Therefore,  a  more  sophisticated  qualitative  analysis  was  needed,  and  a  cluster  analysis  was  conducted  of  the 
interview  data  using  the  commercially  available  "ATLAS.ti"  qualitative  data  analysis  tool.  The  research  question 
for  this  analysis  was:  Is  there  a  common  set  of  practices  that  drive  the  business  model  of  rapid  organizations?  The 
intent  was  to  identify  key  patterns  to  determine  the  most  prevalent  themes  and  the  correlation  of  certain  words. 
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The  hypothesis  was  that  rapid  acquisition  processes  are  governed  by  a  common  set  of  principles.  Via  interview 
sessions  with  these  organizations,  the  expected  outcome  is  an  emergence  of  these  common  governance 
principles.  Potential  second  order  effects  are  that  expedited  SE  and  rapid  acquisition  concepts  could  improve 
processes  for  traditional  programs. 

The  first  round  of  analysis  looked  at  frequency  of  words  and  correlations,  in  total  of  all  interviews.  This  resulted  in 
the  identification  of  12  principles  that  were  documented  by  the  AFIT  master's  students  for  their  research  report. 
(Ford  et  al,  2012)  As  a  result  of  inputs  from  reviewers  during  the  defense  of  their  report,  a  second  round  of 
analysis  was  done  examining  the  frequency  of  words  and  correlation  per  organization. 

A  complete  description  of  the  ATLAS.ti™  data  analysis  methodology  and  results  is  in  Appendix  B. 

The  principles  were  narrowed  down  to  11  direct  observations  of  "Group  1,"  as  described  in  Chapter  2. 


A. 7  Creation  of  the  Framework 

An  original  goal  of  the  RT-34  research  was  to  develop  a  framework  for  scaling  SE  activities  in  response  to  different 
development  constraints,  such  as  reduced  development  time,  with  intent  to  prepare  a  plan  to  validate  the 
framework  on  a  DoD  acquisition  program.  The  framework  evolved  in  an  iterative  process,  using  a  grounded 
theory  approach. 

The  framework  development  began  in  parallel  to  the  data  collection  and  evolved  based  on  the  results  of  the  site 
visit  data  collection,  data  analysis,  the  literature  search,  the  review  of  other  SERC  RTs,  the  review  of  other  known 
MPTs,  and  the  researchers'  knowledge  of  practices  in  other  fields.  As  a  result,  a  framework  began  to  emerge  that 
describes  rapid  development. 

Our  first  framework  was  based  on  the  "lanes  of  acquisition"  as  described  in  Chapter  3.  The  concept  came  about 
early  on,  as  it  appeared  that  the  approach  to  expedited  SE  depended  on  what  product  was  intended  and  what 
systems  engineering  processes  were  tailored  or  scaled  accordingly.  The  product  implied  a  "lane"  of  acquisition, 
and  the  lanes  could  also  be  associated  with  varying  levels  of  systems  engineering  rigor.  We  socialized  this  concept, 
along  with  the  sponsor,  at  the  Conference  on  Systems  Engineering  Research  (CSER)  2012  in  two  separate  plenary 
keynote  presentations,  as  well  as  with  various  discussions  with  SMEs  in  the  community  and  with  the  SERC 
sponsors. 

After  gathering  feedback,  we  realized  that  the  "lanes"  were  a  subset  of  a  larger  framework,  and  not  the  answer 
itself.  (The  lanes  and  flexibility  ultimately  became  a  "Direct  Observation"  of  Group  2.)  The  research  team  met  as  a 
group  for  several  days  in  July  2012  to  debate  this  framework.  The  second  iteration  resulted  in  a  stacked  "triangle¬ 
shaped"  model,  collecting  the  observations  from  the  data  along  with  our  own  interpretation  of  the  data.  This 
implied  a  hierarchy  of  activities  conducted  by  rapid  organizations,  with  increasing  rigor  of  systems  engineering 
and  sophistication.  However,  the  concept  of  a  hierarchy  raised  an  additional  research  question,  as  we  did  not 
have  the  data  to  prove  a  hierarchy.  It  was  this  first  triangle  model  that  was  included  in  the  first  draft  of  the  final 
report  and  distributed  to  all  the  site  visit  organizations  and  individuals  for  feedback,  as  described  earlier. 

We  incorporated  all  the  feedback  received  and  reviewed  the  final  report  draft  2  with  the  sponsor.  Through  this 
preparation  and  discussion,  we  realized  that  some  parts  of  the  framework  described  "what"  organizations  did, 
and  other  parts  described  "how"  they  did  them.  The  framework  should  be  consistent,  and  we  restructured  the 
framework  to  consist  of  "whats"  and  moved  the  "hows"  to  an  appendix  and  described  them  as  enablers.  We 
were  also  challenged  to  think  about  what  would  make  the  framework  useful  to  the  rapid  acquisition  community 
(who  would  say  our  observations  were  "not  new")  and  how  to  make  it  resonate  with  the  traditional  acquisition 
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community  (who  were  typically  biased  against  rapid  programs  because  they  felt  that  rapid  programs  operated 
outside  the  DODI5000  rules).  A  third  iteration,  still  in  a  triangle  shape,  resulted,  along  with  the  shaping  of  future 
research  questions  that  would  address  how  to  expose  and  educate  the  acquisition  community  to  accelerated 
development,  hypothesizing  that  the  experiences  gained  in  a  rapid  program  could  give  engineers  and  contracting 
officers  alike  a  better  understanding  of  a  full  acquisition  cycle  and  the  goal  to  enable  a  warfighter  to  deliver  a 
desired  effect.  This  goes  beyond  just  learning  how  to  follow  a  proscribed  process. 

We  then  shared  this  triangle  shape  with  attendees  to  our  presentation  at  the  NDIA  Systems  Engineering 
Conference  (October  2012).  We  were  looking  for  feedback  on  whether  the  framework  resonated  with  other 
SMEs  in  the  community,  and  whether  we  could  gather  any  insight  to  reasonableness  of  a  hierarchy.  The 
framework  was  also  informally  socialized  in  another  site  visit  with  a  DoD  rapid  cell  that  was  not  part  of  the  initial 
site  visit  group.  Further  feedback  was  gathered  at  the  SERC  Sponsor  Research  Review  in  November  2012  through 
a  SERC  panel.  The  feedback  was  positive,  and  we  kept  the  triangle,  and  documented  the  further  research 
questions  that  it  prompted  (as  described  in  Chapter  5)  such  as  hierarchy,  questions  of  what  was  necessary  and 
sufficient  for  success,  etc. 

The  last  iteration  is  presented  in  this  final  report  and  came  about  from  further  discussions  with  the  RT-34  research 
team  as  well  as  a  "deep  dive"  review  and  discussion  with  SERC  leadership  and  collaborators  from  other  related 
RTs.  We  finally  elected  to  not  infer  any  precedence,  hierarchy,  or  increasing  systems  engineering  rigor  in  the 
framework.  We  eliminated  the  triangle  shape  and  re-ordered  the  groups  to  describe  exactly  what  we  saw  in  this 
research  project. 

For  presentation  in  this  report,  we  break  the  framework  into  three  groups: 

•  Group  1:  Direct  Responses, 

•  Group  2:  Direct  Observations,  and 

•  Group  3:  Inferred  Organizational  Characteristics. 

The  Direct  Responses  in  Group  1  provide  an  efficient  affinity  across  all  answers  to  our  37  questions.  This  set 
represents  the  top  11  "Organizational  Best  Practices".  We  observed  strong  evidence  across  the  site  visits  for  all 
11  observations.  In  the  process  of  validating  the  responses  with  the  SMEs,  reviewing  against  literature,  and 
socializing  the  observations  with  others  in  the  rapid  and  traditional  community,  we  realized  that  the  11 
observations  are  "not  new"  and  do  not  appear  to  be  unique  to  rapid  programs.  We  characterize  these  as 
practices  that  can  be  found  in  both  rapid  and  traditional  development  programs. 

Group  2  "Flexible  Acquisition  Practices"  reflects  Direct  Observations  that  emerged  from  the  site  visits,  but  were 
not  direct  responses  to  the  prepared  questions.  This  group  collects  observations  based  on  flexibility  in  acquisition 
practices,  whether  it  be  contract  type,  finance  type,  program  management  approach,  documentation  required, 
level  of  insight/oversight,  etc.  These  practices  are  seen  as  critical  for  a  rapid  business  environment.  We  originally 
did  not  seek  out  different  types  of  rapid  categories,  or  strictly  acquisition  management  (such  as  contracting) 
findings;  in  fact,  a  ground  rule  of  the  research  was  that  we  would  not  seek  to  change  the  existing  DoD5000 
process.  However,  these  observations  about  the  acquisition  environment  emerged  from  the  site  visits  and  could 
not  be  ignored. 

Group  3,  the  set  of  "Inferred  Organizational  Characteristics,"  represents  aspects  of  a  "Go  Fast"  Culture  seen  in  the 
rapid  organizations.  This  is  where  rapid  organizations  start  to  differ  from  traditional  ones,  and  there  is  a  shift  in 
energy,  commitment,  and  knowledge.  These  findings  are  motivated  by  an  analysis  of  effective  enterprise  culture 
and  business  agility  literature.  The  Group  3  characteristics  include: 
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•  Rapid  organizations  use  intense  and  efficient  knowledge-sharing  to  enable  stabilization  and 
synchronization  of  information 

•  Rapid  organizations  are  characterized  by  a  risk-tolerant  culture 

•  Rapid  organizations  are  structured  for  ambidexterity  with  a  balance  between  exploration  and  exploitation 

•  Rapid  Development  organizations  exercise  Real-time  Management,  with  both  empowerment  at  the 
lowest  level  at  which  sufficient  knowledge  and  experience  exists  to  make  appropriate  decisions  as  well  as 
rapid  access  to  top-level  leadership  to  facilitate  decision  velocity. 

Real-time  management  represents  an  integrative  characteristic;  it  requires  intense  knowledge  sharing,  risk- 
focused  culture  (people  knowing  what  risks  to  take  and  when  and  how),  and  the  organizational  ambidexterity  that 
comes  from  a  balance  of  exploration  and  exploitation. 

The  framework  could  and  likely  will  continue  to  evolve  as  this  final  report  is  socialized  within  the  SERC 
collaborator  and  sponsor  community,  and  as  future  research  questions  are  addressed  in  the  follow-on  activities. 


A. 8  Limitations  and  Assumptions 

The  qualitative  nature  of  this  data  and  grounded  theory  research  allows  for  interpretation  depending  on  the 
readers  or  researchers  point  of  view.  Qualitative  analysis  can  therefore  become  biased  based  on  individual 
experience  and  perspective.  The  research  team  endeavored  to  stay  aware  of  bias  in  guiding  discussions  and 
interpreting  data.  Further,  the  interviews  were  conducted  as  guided  conversations  as  opposed  to  strict  survey 
responses.  The  research  team  ensured  the  topic  of  each  question  was  covered  within  the  course  of  each 
conversation  and  felt  better  data  was  acquired  by  allowing  the  SMEs  to  discuss  the  organization  and  its  processes 
in  their  own  way.  As  noted  in  Section  A.6  above,  errors  could  also  be  introduced  during  the  note-taking  and 
transcription  processes. 

The  data  analysis  with  ATLAS. ti™  tool  analyzed  data  from  a  sub-set  of  the  interviews— covering  22  different 
interviews  and  briefings.  Informal  discussions  and  note  taking  from  conferences  were  not  included  in  this 
analysis.  This  is  because  while  target  of  opportunity  and  short  notice  interviews  afforded  great  openings  for  data 
gathering,  they  did  not  result  in  complete  sets  of  notes.  Further,  not  all  team  members  were  able  to  attend  all 
discussions.  The  first  round  of  data  analysis  was  conducted  by  the  AFIT  student  team,  which  limited  the  data  sets 
to  those  interviews  they  personally  conducted  or  those  with  which  they  had  considerable  background  information 
to  provide  interview  context. 

Many  of  the  organizations  interviewed  were  managing  classified  programs  with  classified  customers.  All 
interviews  were  conducted  in  an  unclassified  environment.  This  may  have  limited  the  extrema  of  detail  provided 
and  potentially  prevented  full  disclosure  of  organizational  practices.  Further,  this  precluded  detailed  discussions 
on  specific  products  these  organizations  have  delivered  or  are  developing  at  the  time  of  the  interview. 

Discussions  held  in  secure  facilities  did  not  permit  recording  devices,  so  all  notes  were  hand-written,  which  could 
introduce  biases  as  described  earlier. 

As  a  part  of  the  literature  search,  various  Systems  Engineering  case  studies  from  both  traditional  and  rapid 
organizations  were  analyzed  in  an  attempt  to  map  real-world  data  back  to  the  proposed  framework.  This  proved 
to  be  difficult,  as  existing  case  study  data  (such  as  AFIT  published  case  studies)  were  not  geared  to  address  the 
question  of  success  according  to  this  framework.  Further,  the  RT-34  interviews  themselves  did  not  specifically  ask 
whether  and  how  certain  principles  resulted  in  success  or  failure.  The  questions  assumed  that  the  rapid 
organizations  were  successful  in  what  they  did.  Delving  into  these  questions  would  require  collecting  data  from 
specific  programs  or  projects  in  which  the  definition  and  metrics  of  success  or  failure  could  be  further  explored. 
This  is  an  area  for  future  research. 
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A. 9  Summary 

RT-34  provides  an  initial  basis  to  understand  U.S.  military  urgent  needs  programs  and  how  they  conduct  rapid 
development.  RT-34  hypothesized  that  rapid  acquisition  organization  success  is  governed  by  a  common  set  of 
principles  and  that  via  interview  sessions  with  these  organizations,  common  governance  would  emerge.  Through 
the  analysis,  a  common  set  of  principles  did  emerge  and  attempts  are  already  being  made  to  apply  some  of  them 
to  traditional  acquisition  programs. 

It  is  an  underlying  assumption  that  the  organizations  interviewed  achieved  success  in  some  right,  whether  that  be 
cost,  schedule,  or  delivering  a  product  to  the  field.  Attempts  were  not  made  to  explicitly  define  success  in  these 
organizations,  but  rather  assume  that  by  their  very  nature  and  existence,  they  are  successful  in  some  way.  While 
the  original  hypothesis  sought  to  identify  critical  success  factors,  the  interviews  and  data  analysis  have  revealed 
principles  and  common  themes,  but  not  whether  the  themes  would  be  necessary  or  sufficient  for  success.  This  is 
an  area  for  further  research. 


Table  A-l:  List  of  Site  Visits  Conducted 


1 

The  Aerospace  Corporation  Concept  Design  Center  (CDC),  Los  Angeles,  CA 

2 

American  Institute  of  Aeronautics  and  Astronautics  (AIAA)  Space  2011  conference,  Long  Beach,  CA  (various  discussions 
with  SMEs  from  government,  industry,  and  academia) 

3 

An  aerospace  industry  innovation  lab,  Southern  California 

4 

A  rocket  engine  design  company,  Southern  California 

5 

Annual  SERC  Research  Review  (ASRR),  College  Park,  MD  (various  discussions  and  presentation  notes,  including  a  panel 
discussion  on  this  research  subject) 

6 

Small  satellite  development  company 

7 

NASA  Goddard  Space  Flight  Center  (GSFC),  Greenbelt,  MD  (multiple  separate  discussions) 

8 

Operationally  Responsive  Space  (ORS)  Office,  Kirtland  AFB,  NM  (multiple  discussions) 

9 

USAF  Space  Development  and  Test  Directorate,  Kirtland  AFB,  NM  (multiple  discussions) 

10 

Int'l  Symposium  for  Personal  &  Commercial  Spaceflight  (ISPCS)  (several  discussions  w/speakers  &  attendees) 

11 

International  Women's  Forum  Annual  Conference,  Washington,  DC  (notes  from  panel  presentation) 

12 

Air  Force  Rapid  Capabilities  Office  (RCO),  Washington  DC 

13 

Intelligence  Community,  Washington  DC  (multiple  organizations  represented) 

14 

Georgia  Tech  Research  Institute  (GTRI)  Collaborative  Visualization  Environment  ("COVE"),  Atlanta,  GA 

15 

U.S.  Army  Prototype  Integration  Facility  (PIF),  Huntsville,  AL 

16 

Aerospace  /  Engineering  small  business,  Huntsville,  AL  (multiple  discussions) 

17 

Big  Safari  -  Program  office  and  AF  Program  Executive  Office,  ISR-SOF,  Wright-Patterson  AFB  OH 

18 

Air  Force  Research  Lab  (AFRL)  Center  for  Rapid  Product  Development  (CRPD),  Wright-Patterson  AFB,  OH 

19 

AF  Aeronautical  Systems  Center,  Rapid  Development  Integration  Facility  (RDIF),  Wright-Patterson  AFB,  OH 

20 

AFRL  Air  Vehicles  Directorate,  Wright-Patterson  AFB,  OH  (multiple  discussions) 

21 

Air  Force  Space  Command  /  A5,  Peterson  AFB,  CO 

22 

Air  Force  Space  and  Missile  Systems  Center,  Rapid  Reaction  Branch,  Colorado  Springs,  CO 

23 

Air  Force  Tactical  Exploitation  of  National  Capabilities  (TENCAP),  Schriever  AFB,  CO 

24 

Air  Force  Space  and  Missile  Systems  Center  HQ,  Los  Angeles,  CA 

25 

SOCOM  Research  and  Development  Acquisition  Center  (SORDAC),  MacDill  AFB,  FL 
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ATTACHMENT  1:  ORIGINAL  LIST  OF  EXPEDITED  SE  QUESTIONS 

RT-34  Questions  -  September  2011.  The  following  introduction  and  list  of  questions  was  provided  to  site  visits 
in  LA  on  September  26  and  28,  2011. 


Systems  Engineering  Research  Center 
Research  Topic  #34  (RT-34) 
Expedited  Systems  Engineering 
Site  Visits  in  LA,  Sept  26  and  28,  2011 


RT-34  Problem  Statement 

The  Defense  Science  Board  (DSB)  Task  Force  on  the  Fulfillment  of  Urgent  Operational  Needs  (July  2009)  identified 
more  than  20  rapid-reaction  programs  and  organizations  addressing  DoD  urgent  warfighter  needs.  A  subsequent 
DSB  study  found  that  urgent  needs  programs  spent  more  than  $50  billion  between  2005-2009,  and  that  urgent 
needs  should  be  considered  a  critical,  ongoing  DoD  institutional  capability.  In  other  words,  "urgent"  is  becoming 
more  "normal."  In  March  2011,  the  GAO  report,  "DoD's  Urgent  Needs  Processes  Need  a  More  Comprehensive 
Approach  and  Evaluation  for  Potential  Consolidation,"  found  that  DoD  has  taken  steps  to  improve  fulfillment  of 
urgent  needs,  but  it  needs  a  common  approach  for  addressing  urgent  needs.  The  GAO  report  refers  to  at  least  31 
entities  that  manage  urgent  needs  and  expedite  the  solutions  to  address  them. 

This  SERC  project  will  examine  expedited  systems  engineering  as  applied  to  rapid  capability  and  urgent  needs  as 
developed  in  response  to  changing  threats.  Lifecycle  of  urgent  needs  programs  is  driven  by  "time  to  market"  as 
opposed  to  complete  satisfaction  of  static  requirements,  with  delivery  expected  in  days/months  versus 
years/decades.  The  processes  and  practices  applied  to  urgent  needs  must  add  value  and  not  require  an  excessive 
bureaucratic  oversight  to  implement,  while  at  the  same  time  address,  understand,  and  manage  risk  such  that 
programs  can  understand  better  where  to  include,  truncate,  or  eliminate  systems  engineering  (SE)  practices  and 
processes.  The  hypothesis  is  that  by  defining,  identifying,  testing,  and  ultimately  implementing  expedited  SE 
processes  and  practices,  capability  that  results  from  urgent  needs  may  be  more  effective,  efficient,  and  longer 
lasting  in  the  field.  Potential  second  order  effects  are  that  expedited  SE  as  applied  to  urgent  needs  could 
streamline  specific  future  SE  practices,  as  urgent  becomes  "normal,"  and  findings  could  eventually  improve  SE 
processes  for  traditional  programs  as  well. 

The  purpose  of  the  research  is  to  explore  and  develop  a  scalable  expedited  SE  framework  for  hybrid  programs,  i.e., 
those  utilizing  rapid  acguisition  procurement  but  with  the  intent  to  have  a  more  traditional  lifecycle  for 
deployment,  maintainability,  reliability,  adaptability  and  sustainment.  The  framework  will  examine  scaling  ofSE 
activities  in  response  to  different  development  constraints,  such  as  reduced  development  time. 

RT-34  is  a  9-month  research  task,  from  September  1,  2011  -  May  31,  2012. 

Phase  1:  Short  planning  phase  to  identify  organizations  practicing  expedited  systems  engineering, 
including  visiting  selected  organizations  and  incorporating  input  from  the  SERC  Research  Council. 

Phase  2:  Analyze  current  state  of  the  art  in  expedited  systems  engineering  within  DoD  and  commercial 
sector,  developing  a  framework  for  scaling  SE  activities  in  response  to  different  development  constraints, 
such  as  reduced  development  time. 

Phase  3:  Prepare  a  plan  for  validating  the  framework  on  a  DoD  acquisition  program. 

Phase  4  (Future,  Separate  Funding):  Execute  the  plan  from  Phase  3,  with  research  to  analyze  the 
framework  in  action  and  iterate  it  based  on  observations  and  results  as  applied  to  a  real  program. 
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Site  Visits  to  Aerospace  Corporation  Concept  Design  Center  &  Industry  Innovation  Lab 

Context:  Both  participated  in  a  previous  research  paper  [Jo  Ann  Lane  et  al]  on  "Critical  Success  Factors  for  Rapid, 
Innovative  Solutions,"  which  posed  a  series  of  questions  on  scope,  processes,  methods,  product,  tools,  people, 
work  space,  and  key  success  factors.  The  paper  concluded  that  successful  innovative  organizations  share  certain 
characteristics: 

•  Driven  by  business  value 

•  Take  calculated  risks 

•  Follow  concurrent  engineering  practices 

•  Look  for  solution  patterns  that  can  be  re-used 

•  Proactive  management  and  small  agile  teams 

•  Provide  culture  and  environment  that  supports  innovation 

RT-34  examines  the  systems  engineering  side  of  rapid  development. 
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RT-34  Questions  (as  of  9/26/11) 


1. 


2. 


3. 


4. 


5. 

6. 


7. 


8. 


What  systems  engineering  methods,  processes,  and  tools  do  you  use 

a.  What  is  considered  "standard"  in  your  organization 

b.  If  processes  are  tailored,  which  ones  are  tailorable  and  why;  how  do  you  capture  this  knowledge 

c.  What  rationale  is  used  to  follow  a  standard  process  or  not 

d.  What  is  the  role  of  the  project  manager,  chief  engineer,  chief  architect,  senior  leadership,  customer 

e.  What  level  of  risk  is  acceptable  and  how  do  you  determine 

f.  What  level  of  documentation  do  you  use  and  why 

g.  What  types  of  meetings  do  you  hold,  who  attends,  who  makes  decisions,  and  why 

h.  How  replicable  /  transferable  are  your  processes  from  one  project  to  another  and  why 
Scope 

a.  How  long  is  the  lifecycle  of  the  product 

b.  How  many  units  are  you  supporting 

c.  What  different  MPTs  do  you  employ  based  on  scope 
Team 

a.  What  types  of  teams  do  you  use  (e.g.,  domain,  functional,  IPT,  etc) 

b.  How  do  you  select  the  team 

c.  Who  selects  the  team  members  and  why 

d.  What  skill  mixes  are  the  best 

e.  How  homogeneous  (or  not)  are  the  skills,  people,  and  personalities 

f.  How  do  you  bring  in  the  government  sponsor 

g.  How  do  you  bring  in  the  user  perspective 

h.  How  do  you  manage  people  and  teams  that  are  not  co-located 

i.  How  do  you  network  across  these  boundaries 

j.  How  do  you  handle  round-the-clock  work  and  burnout,  what  is  the  maximum  timeframe  that  people 
can  withstand  the  stress  (role  of  stress) 

Decision  Analysis 

a.  How  are  systems  engineering  decisions  made 

b.  Who  makes  the  decisions 

c.  At  what  level  are  decisions  made 

d.  Who  is  empowered,  how  do  they  know  it,  how  are  they  supported 

e.  Is  there  a  chain  of  command  and  how  is  it  used 

f.  How  do  you  know  when  and  where  to  take  risks 

g.  To  what  extent  are  these  decisions  documented,  formalized,  communicated 

h.  How  do  you  prepare  for  decisions 

i.  How  quickly  do  you  need  to  make  decisions 

j.  How  is  decision  making  in  the  critical  path 
Product  Use 

a.  How  do  you  translate  prototypes  to  operational  use 
Scalability 

a.  How  are  your  responses  above  dependent  on  size  of  the  project  (scope,  cost,  timeline,  risk,  #  of 
people) 

b.  What  systems  engineering  tasks  are  tailorable  or  scalable? 

Collaboration  and  Communication 

a.  What  role  does  collaboration  play.. .in  general,  in  management,  in  team  building,  in  problem  solving, 
in  systems  engineering  processes,  in  communication,  in  knowledge  transfer 

b.  How  do  you  facilitate  collaboration  (internal,  external) 

What  collaborative  tools  do  you  use,  which  are  most  effective  in  which  situation  and  why 
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ATTACHMENT  2:  FINAL  LIST  OF  EXPEDITED  SE  QUESTIONS 

RT-34  Questions  (as  of  10/10/2011).  The  following  was  provided  to  site  visits  conducted  after  October  10, 
2011. 


Systems  Engineering  Research  Center  (SERC) 
Research  Topic  34  (RT-34) 
Expedited  Systems  Engineering 
Site  Visits:  September  2011  -  April  2012 


RT-34  Problem  Statement 

The  Defense  Science  Board  (DSB)  Task  Force  on  the  Fulfillment  of  Urgent  Operational  Needs  (July  2009)  identified 
more  than  20  rapid-reaction  programs  and  organizations  addressing  DoD  urgent  warfighter  needs.  A  subsequent 
DSB  study  found  that  urgent  needs  programs  spent  more  than  $50  billion  between  2005-2009,  and  that  urgent 
needs  should  be  considered  a  critical,  ongoing  DoD  institutional  capability.  In  other  words,  "urgent"  is  becoming 
more  "normal."  In  March  2011,  the  GAO  report,  "DoD's  Urgent  Needs  Processes  Need  a  More  Comprehensive 
Approach  and  Evaluation  for  Potential  Consolidation,"  found  that  DoD  has  taken  steps  to  improve  fulfillment  of 
urgent  needs,  but  it  needs  a  common  approach  for  addressing  urgent  needs.  This  2011  GAO  report  documents  at 
least  31  entities  that  manage  urgent  needs  and  expedite  the  solutions  to  address  them.  The  Systems  Engineering 
Research  Center  (SERC),  a  DoD  University  Affiliated  Research  Center  (UARC),  has  been  funded  by  the  Air  Force 
(SAF/AQR)  to  conduct  research  exploring  this  problem. 

The  SERC  project  on  Expedited  Systems  Engineering  (SE)  will  examine  expedited  systems  engineering  as  applied  to 
rapid  capability  and  urgent  needs  as  developed  in  response  to  changing  threats.  Lifecycle  of  urgent  needs 
programs  is  driven  by  "time  to  market"  as  opposed  to  complete  satisfaction  of  static  requirements,  with  delivery 
expected  in  days/months  versus  years/decades.  The  successful  techniques  seen  in  rapid  prototyping  must  scale 
to  larger,  more  complex,  and  supportable  weapon  systems.  The  engineering  processes  and  practices  applied  to 
urgent  needs  must  innovate  conceptual  solutions,  quickly  prune  the  design  space,  and  choose  good  designs  that 
can  delivery  warfighting  capability.  The  hypothesis  is  that  by  defining,  identifying,  testing,  and  ultimately 
implementing  expedited  SE  processes  and  practices,  capability  that  results  from  urgent  needs  may  be  more 
effective,  efficient,  and  longer  lasting  in  the  field.  Potential  second  order  effects  are  that  expedited  SE  as  applied 
to  urgent  needs  could  streamline  specific  future  SE  practices,  as  urgent  becomes  "normal,"  and  findings  could 
eventually  improve  SE  processes  for  traditional  programs  that  want  to  operate  more  rapidly. 

The  purpose  of  the  research  is  to  explore  and  develop  a  scalable  expedited  SE  framework  for  hybrid  programs,  i.e., 
those  exploiting  rapid  development,  but  with  the  intent  to  have  traditional  lifecycle  considerations  for  deployment, 
maintainability,  reliability,  adaptability  and  sustainment.  Likewise,  this  new  SE  framework  would  be  applicable  to 
more  traditional  acguisition  programs  with  a  desire  to  incorporate  scaled  rapid  development  best  practices.  The 
focus  is  on  systems  engineering,  not  acguisition  processes. 

RT-34  is  a  9-month  research  task,  from  September  1,  2011  -  May  31,  2012. 

•  Phase  1:  Short  planning  phase  to  identify  organizations  practicing  expedited  systems  engineering, 
including  visiting  selected  organizations  and  incorporating  input  from  the  SERC  Research  Council. 

•  Phase  2:  Analyze  current  state  of  the  art  in  expedited  systems  engineering  within  DoD  as  well  as  non-DOC 
and  commercial  sector,  developing  a  framework  for  scaling  SE  activities  in  response  to  different 
development  constraints,  such  as  reduced  development  time. 

•  Phase  3:  Prepare  a  plan  for  validating  the  framework  on  a  DoD  acquisition  program. 
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•  Phase  4  (Future,  Separate  Funding):  Execute  the  plan  from  Phase  3,  with  research  to  analyze  the 
framework  in  action  and  iterate  it  based  on  observations  and  results  as  applied  to  a  real  program. 


Context:  A  recent  research  paper  [Jo  Ann  Lane,  et  al]  on  "Critical  Success  Factors  for  Rapid,  Innovative  Solutions," 
posed  a  series  of  questions  on  potential  program  success  factors.  That  paper  concluded  that  successful  innovative 
organizations  share  certain  characteristics: 

•  Driven  by  business  value  • 

•  Take  calculated  risks  • 

•  Proactive  management  and  small  agile  • 

teams 


Look  for  solution  patterns  that  can  be  re-used 
Follow  concurrent  engineering  practices 
Provide  culture  and  environment  that  supports 
innovation. 
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RT-34  Questions  (as  of  10/10/2011) 

1.  Systems  engineering  (SE)  methods,  processes,  and  tools 

a.  Do  you  use  standard/  formal  SE  processes  in  your  rapid  development  organizations?  Which  ones? 

b.  Are  SE  processes  tailored  for  each  program  or  product.  If  so,  which  ones  can  be  highly  tailorable  and 
why 

c.  How  are  SE  methods,  processes  and  tools  different  based  on  project  scale/  scope 

d.  What  level  of  risk  is  acceptable,  how  do  you  determine  that,  and  how  do  you  systemically  address  it 
at  all  levels 

e.  What  is  the  formality  of  engineering  documentation 

f.  How  replicable  /  transferable  are  your  processes  from  one  project  or  product  to  another 

g.  How  do  model-based  systems  engineering  approaches  support  your  rapid  development 

h.  Do  you  integrate  a  variety  of  models/  simulations/  prototypes  early  in  the  lifecycle,  and  if  so,  how 

i.  How  would  you  describe  your  ability  to  be  innovative  in  concept  refinement 

j.  What  are  best  practices  for  problem  domain  understanding 

k.  How  do  you  manage  scope  and  requirements 

l.  What  infrastructure  (tools,  modeling  &  sim)  allows  continuously  quickening  product  delivery  cycles 

m.  Decision  Analysis  Processes 

i.  Who,  and  at  what  level,  are  most  engineering  decisions  made 

ii.  Who  is  empowered,  how  do  they  know  it,  how  are  they  supported 

iii.  To  what  extent  are  major  decisions  documented,  formalized,  communicated 

iv.  How  do  you  prepare  for  major  decisions 

v.  What  role  does  decision  making  play  in  the  critical  path 

2.  People/Team/Collaboration 

a.  What  types  of  teams  do  you  use  (e.g.,  domain,  functional,  IPT,  etc) 

b.  What  are  the  primary  leadership  roles  for  an  expedited  project  or  for  the  best  projects  that  run  the  most 
efficiently  (program  or  project  manager,  chief  engineer,  chief  architect,  etc) 

c.  How  do  you  select/ design  the  team 

d.  What  are  the  primary  skills  you  seek  for  the  team 

e.  How  do  you  effectively  incorporate/involve  the  end  user 

f.  How  do  you  effectively  and  continuously  incorporate  the  user  perspective 

g.  How  do  you  manage  and  network  people  and  teams  that  are  not  co-located 

h.  What  role  does  collaboration  play...  in  management,  in  team  building,  in  problem  solving,  in  SE 
processes,  and  in  geographically  distributed  teams 

i.  How  do  you  facilitate  improved  collaboration  (internal,  external) 

j.  What  collaborative  tools  or  processes  do  you  use 

k.  What  types  of  meetings  do  you  hold,  who  attends,  who  makes  decisions,  and  why 

l.  How  do  you  manage  urgent  project  tempos  and  its  personnel  effects  (stress,  work  hours,  burnout) 

m.  How  do  you  reduce  complexity  of  the  SE  process 

3.  Architecture  /  Product 

a.  How  do  you  translate  prototypes  to  operational  use 

b.  How  long  is  the  intended  operational  lifecycle  of  the  product 

c.  How  many  units  are  you  producing/fielding 

d.  How  does  your  rapid  development  schedule  drive  architectural/  design  choices 

e.  How  does  reuse,  modification  of  existing  systems,  or  using  product  lines  drive  reduced  schedules 

f.  How  does  the  level  of  complexity  effect  the  product  architecture 

4.  Scalability  -  How  are  your  responses  dependent  on  size  of  the  project  (scope,  cost,  timeline,  risk, 
number  of  people) 
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Input 

Input  is  welcome  on  this  research,  in  the  form  of  answers  to  these  questions  (from  individuals  or  programs) 
as  well  as  ideas  for  DoD  acquisition  programs  (Phase  3  and  future  Phase  4  funding)  that  could  serve  as 
testbeds  for  the  expedited  framework. 


RT-34  Points  of  Contact 

Ms  Debra  Facktor  Lepore 
Principal  Investigator  (PI),  SERC  RT-34 
Stevens  Institute  of  Technology 
debra.lepore(5)stevens.edu 

201-216-8103  (office) 

425-985-1350  (cell) 


Dr  John  Colombi 

Co-PI,  SERC  RT-34 

Air  Force  Institute  of  Technology 

John.colombi(5)afit.edu 

937-255-3355  x3347  (office) 


Systems  Engineering  Research  Center  (SERC) 

www.SERCuarc.org 

The  Systems  Engineering  Research  Center  (SERC)  is  a  University  Affiliated  Research  Center  (UARC),  competitively 
awarded  by  the  U.S.  Department  of  Defense  to  Stevens  Institute  of  Technology  in  2008.  The  SERC  leverages  the 
research  and  expertise  of  senior  lead  researchers  from  over  20  collaborator  universities  and  not-for-profit  research 
organizations  throughout  the  United  States,  with  more  than  300  researchers  working  on  over  30  tasks  over  the  past 
3  years.  SERC  researchers  have  worked  with  a  wide  variety  of  domains  and  industries,  and  so  are  able  to  bring  views 
and  ideas  from  beyond  the  traditional  defense  industrial  base.  The  SERC  is  sponsored  by  ASD(R&E)  and  includes 
strategic  sponsors  such  as  the  Defense  Acquisition  University,  the  U.S.  Army,  and  the  U.S.  Air  Force.  Through  its 
collaborative  research  concept,  the  SERC  embodies  the  potential  to  radically  improve  the  application  of  systems 
engineering  to  the  successful  development,  integration,  testing  and  sustainability  of  complex  systems,  services  and 
enterprises.  The  vision  is  to  be  the  networked  national  resource  to  further  systems  research  and  its  impact  on  issues 
of  national  and  global  significance. 
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Appendix  B:  Data  Analysis 


B.l  Analysis  Methodology 

Using  the  interview  notes,  a  qualitative  analysis  was  performed  using  ATLAS.ti™  software  in  an  effort  to  further 
explore  the  data  for  hidden  connections  and  emergent  trends.  Full,  transcribed  interview  notes,  from  22  of  the  25 
site  visits  were  assigned  to  an  ATLAS.ti™  hermeneutic  unit.  As  described  in  Appendix  A,  target  of  opportunity  and 
shorter  informal  discussions  at  conferences  afforded  great  openings  for  data  gathering,  but  this  analysis  described 
below  was  limited  to  full  site  visits.  Using  the  11  common  expedited  practices  (Chapter  2)  derived  through  the  site 
visit  discussion  process  as  codes,  each  document  was  coded  appropriately  against  key  words  and  phrases.  The  codes 
were  organized  into  three  families  of  (1)  System/Product,  (2)  Process  and  Tools,  and  (3)  People  and 
Organization/Governance.  Finally,  an  analysis  was  performed  to  build  theory  regarding  the  relationships  between 
the  common  practices  and  derive  conjecture  regarding  which  practices  may  be  more  prevalent  across  different  rapid 
organizations  and  domains. 


B.2  Data  Analysis  Results 

While  the  22  sampled  organizations  in  the  data  analysis  create  different  products  and  use  different  processes, 
several  common  and  noteworthy  themes  emerged  from  the  data.  During  the  qualitative  software  analysis,  the  11 
practices  discussed  previously  were  coded  a  total  of  310  times  against  the  transcribed  interview  notes.  Figure  B-l 
shows  the  number  of  times  each  specific  practice  was  materially  mentioned  or  cited  as  a  significant  or  important 
business  practice  in  the  discussion  notes. 


Figure  B-l:  Total  Practice  Citation  Counts  from  Sampled  Site  Visit  Data 
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The  data  presented  in  Figure  B-l  shows  a  dominant  trend  of  the  top  five  most  common  occurring  practices  of  rapid 
organizations,  based  solely  upon  the  initial  qualitative  coding  analysis.  These  five  of  the  11  total  practices  identified 
comprised  more  than  two-thirds  (64.2%)  of  the  total  citations  made  during  the  analysis,  with  199  out  of  310  total 
codes,  and  three  of  the  top  five  originating  in  the  People  and  Organization/Governance  domain.  See  Table  B-l. 

Table  B-l:  Top  5  Practices  as  Mentioned  by  Total  Citation 


Top  5  Practices 

Practice 

Domain 

Citations 

1. 

Build  and  Maintain  Trust 

People  &  Organization/Governance 

45 

2. 

Defined  Set  of  Stable  Requirements  focused  on  Warfighter  Needs 

Process  &  Tools 

44 

3. 

Populate  Your  Team  with  Specific  Skills  and  Experience 

People  &  Organization/Governance 

40 

4. 

Maintain  High  Levels  of  Motivation  and  Expectations 

People  &  Organization/Governance 

36 

5. 

Work  to  Exploit  Maximum  Flexibility  Allowed 

Process  &  Tools 

34 

Examining  the  number  of  citations  does  have  some  limitations,  as  the  database  contained  transcribed  hand-written 
notes  from  each  discussion.  These  limitations  include  the  following:  organizations  or  individuals  may  use  different 
words  to  describe  the  same  thing;  the  same  word  or  collection  of  words  may  be  heard  multiple  times  by  the 
researcher  and  thus  not  written  down  any  more  (because  that  thought  was  "captured"  already);  the  researcher 
could  write  down  his  or  her  own  interpretation  of  a  word  or  concept. 

A  limitation  of  this  approach  is  that  the  number  of  total  citations  could  be  influenced  by  one  organization  citing  one 
area  multiple  times.  To  explore  this  bias,  a  second  analysis  was  conducted  to  examine  citations  per  organization. 

The  coding  of  practices  to  interview  data  did  not  result  in  every  organization  receiving  a  corresponding  citation  for 
each  of  the  11  practices.  While  many  of  the  practices  were  noted  in  almost  all  the  organizations,  several  practices 
were  only  prevalent  in  a  small  majority  of  organizations,  and  some  organizations  were  noted  as  emphasizing  or 
mentioning  a  specific  practice  multiple  times,  resulting  in  multiple  codes  against  that  practice.  For  comparison, 
Figure  B-2  shows  the  total  number  of  organizations  citing  each  practice  at  least  once. 


Figure  B-2:  Number  of  Organizations  Citing  each  Practice  at  least  once 
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To  further  examine  the  occurrence  of  practices  by  each  organization,  the  average  occurrence  of  each  practice  across 
all  organizations  was  examined.  The  two  practices  with  the  highest  average  occurrence  (defined  as  the  average  time 
each  organization  was  coded  against  that  practice)  are  Build  and  Maintain  Trust  and  Defined  Set  of  Stable 
Requirements  focused  on  Warfighter  Needs;  as  expected  based  upon  the  top  five  occurring  practices  presented 
above,  occurring  on  average  twice  in  each  organization.  For  contrast,  Figure  B.3  shows  the  average  number  of  times 
each  practice  was  coded  across  all  organizations  and  the  maximum  time  each  practice  occurred  in  any  organization. 


Use  Mature 
Technology&Redu 
ce  Complexity- 
Focus  on  the  State 
of  the  Possible 


Right-sizing  the 
Product.. .Eliminat 
es  Major  Prograi 
Oversight 


Incremental 
Deployment 
(Development)  is 
Part  of  the 
oductPlan 


Government 
Team  Leads  the 

Way 


Maintain  High 
Levels  of 
Motivation  and 
Expectations 


Populate  Vour 
Team  with 
Specific  Skills  and 
Experience 


Defined  Set  of 
Stable 

Requirements 
focused  on 
Warfighter  Needs 


Work  to  Exploit 
Maximum 
Flexibility  Allowed 


esigningOut  All 
Risk  takes 
Forever.. .Accept 
Some  Risk 


Maintain  Trust 


eep  an  Eye  on 
Normalization 


- AVERAGE  NUMBER  OF  TIMES  CODED  ACROSS  ALL  ORGANIZATIONS 

- MAXOCCURANCE  BY  ANY  ORGANIZATION 

Figure  B-3:  Average  and  Maximum  Occurrence  of  Practices  by  any  One  Organization 

Further  analysis  was  done  to  examine  co-occurrences  between  practices.  Co-occurrence  is  a  tool  to  examine  where 
potential  relationships  or  interdependencies  exist  in  an  effort  to  discover  which  principles  may  relate  to  one 
another.  In  the  context  of  the  analysis,  practices  that  were  coded  to  the  same  document  within  the  corpus  of  text 
(semantic  proximity)  generate  a  co-occurrence.  For  example,  if  a  segment  of  the  transcribed  notes  from  an  interview 
was  coded  to  the  practice  "Build  and  Maintain  Trust"  within  the  document,  and  then  another  portion  of  the 
document  was  coded  with  "Work  to  Exploit  Maximum  Flexibility  Allowed";  a  co-occurrence  was  generated  between 
the  two  practices.  The  raw  co-occurrences  between  practices  were  then  normalized  by  the  total  number  of  co¬ 
occurrences.  The  results  in  Table  B.2  show  the  highest  probability  of  co-occurrence  in  within  the  context  of  the 
transcribed  documents  between  the  practice  of  "Build  and  Maintain  Trust"  and  two  other  practices:  "Defined  Set  of 
Stable  Requirements  focused  on  Warfighter  Needs"  and  "Populate  Your  Team  with  Specific  Skills  and  Experience." 
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Table  B-2:  Co-occurrence  Matrix 


SYSTEM/  PRODUCT 

PROCESS  &  TOOLS 

PEOPLE  &  ORGANIZATION/GOVERNANCE 

Use  Mature  Technoiogy&Reduce  Complexity-Focus 

on  the  State  of  the  Possible 

Incremental  Deployment  (Development)  is  Part  of 

the  Product  Plan 

Defined  Set  of  Stable  Requirements  focused  on 

Warfighter  Needs 

Work  to  Exploit  Maximum  Flexibility  Allowed 

Designing  Out  All  Risk  takes  Forever. ..Accept  Some 

Risk 

Keep  an  Eye  on  "Normalization" 

Build  and  Maintain  Trust 

Populate  Your  Team  with  Specific  Skills  and 

Experience 

Maintain  High  Levels  of  Motivation  and  Expectations 

Government  Team  Leads  the  Way 

Right-sizing  the  Product. ..Eliminates  Major  Program 

Oversight 

SYSTEM/ 

PRODUCT 

Use  Mature  Technoiogy&Reduce  Complexity- 
Focus  on  the  State  of  the  Possible 

0.012939 

0.012939 

0.010166 

0.009242 

0.009242 

0.013863 

0.011091 

0.010166 

0.005545 

0.005545 

Incremental  Deployment  (Development)  is 
Part  of  the  Product  Plan 

0.012939 

\ 

0.013863 

0.009242 

0.010166 

0.011091 

0.014787 

0.012015 

0.010166 

0.005545 

0.005545 

PROCESS  &  TOOLS 

Defined  Set  of  Stable  Requirements  focused 
on  Warfighter  Needs 

0.012939 

0.013863 

0.012939 

0.010166 

0.010166 

0.015712 

0.013863 

0.012939 

0.00647 

0.00647 

Work  to  Exploit  Maximum  Flexibility  Allowed 

0.010166 

0.009242 

0.012939 

0.008318 

0.009242 

0.013863 

0.012015 

0.009242 

0.00647 

0.004621 

Designing  Out  All  Risk  takes  Forever. ..Accept 
Some  Risk 

0.009242 

0.010166 

0.010166 

0.008318 

0.007394 

0.010166 

0.008318 

0.007394 

0.003697 

0.003697 

Keep  an  Eye  on  "Normalization" 

0.009242 

0.011091 

0.010166 

0.009242 

0.007394 

0.012015 

0.009242 

0.00647 

0.005545 

0.002773 

PEOPLE  &  ORGANIZATION/  GOVERNANCE 

Build  and  Maintain  Trust 

0.013863 

0.014787 

0.015712 

0.013863 

0.010166 

0.012015 

0.015712 

0.013863 

0.007394 

0.004621 

Populate  Your  Team  with  Specific  Skills  and 
Experience 

0.011091 

0.012015 

0.013863 

0.012015 

0.008318 

0.009242 

0.015712 

\ 

0.012939 

0.00647 

0.004621 

Maintain  High  Levels  of  Motivation  and 
Expectations 

0.010166 

0.010166 

0.012939 

0.009242 

0.007394 

0.00647 

0.013863 

0.012939 

\ 

0.005545 

0.004621 

Government  Team  Leads  the  Way 

0.005545 

0.005545 

0.00647 

0.00647 

0.003697 

0.005545 

0.007394 

0.00647 

0.005545 

0.001848 

Right-sizing  the  Product. ..Eliminates  Major 
Program  Oversight 

0.005545 

0.005545 

0.00647 

0.004621 

0.003697 

0.002773 

0.004621 

0.004621 

0.004621 

0.001848 

The  results  of  this  co-occurrence  matrix  are  somewhat  intuitive.  Practices  that  were  coded  more  often  against  a 
transcribed  interview  tend  to  show  more  co-occurrence,  as  can  be  seen  when  comparing  the  matrix  above  with  the 
top  five  most  occurring  practices.  However,  the  avenues  of  analysis  presented  here  begin  to  show  some  emergent,  if 
not  instinctive,  correlations.  For  example,  the  practices  of  "Use  Mature  Technology  &  Reduce  complexity-Focus  on 
the  State  of  the  Possible"  and  "Incremental  Deployment  (Development)  is  Part  of  the  Product  Plan"  both  showed 
strong  co-occurrence  with  "Defined  Set  of  Stable  Requirements  focused  on  Warfighter  needs." 

This  correlation  may  imply  that  rapid  organizations  tend  to  focus  requirements  directly  upon  what  can  be 
accomplished  in  the  near  term  while  achieving  at  least  some  sort  of  solution  to  the  problem,  with  the  intent  of 
upgrading  later  as  required— which  is  arguably  a  common  tenant  among  smaller,  rapid  Government  acquisition 
organizations.  Furthermore,  practices  under  the  People  and  Organization/Governance  family  showed  the  most  co¬ 
occurrence  with  other  practices. 
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Combined  with  the  analysis  showing  three  of  the  five  most  commonly  occurring  practices  under  this  same  domain 
(see  above),  an  implication  is  apparent  that  the  principles  that  define  the  people  on  a  team  have  significant 
contribution  to  the  overall  performance  of  the  effort.  In  the  context  of  the  analysis,  if  an  organization  had  the  right 
people  supporting  a  program,  it  appears  they  are  more  likely  to  keep  the  process  in  check  to  get  the  right  system  or 
product  delivered. 

As  described  in  Chapter  3,  the  organizations  interviewed  appeared  to  focus  and  operate  within  four  lanes  of 
acquisition.  Because  the  organizations  varied  widely  in  size  and  scope,  the  final  portion  of  the  data  analysis  focused 
on  examining  the  importance  of  each  practice  to  the  organizations  within  the  three  rapid  lanes  of  acquisition.  To 
determine  importance,  a  standardization  scheme  was  derived  to  assign  a  scaled  importance  factor  to  each  practice 
with  the  organizations.  Because  the  11  practices  were  the  result  of  emergent  trends  from  the  qualitative  analysis  of 
the  entirety  of  the  interview  data,  it  was  assumed  that  every  practice  holds  at  least  the  minimum  importance  factor 
value  of  1.  The  importance  of  each  practice  was  classified  from  1  to  4,  based  upon  how  many  times  the  particular 
practice  was  coded  against  an  organization,  under  the  assumption  that  multiple  citations  of  a  practice  to  an 
organization  indicates  an  increased  significance  to  the  hallmarks  of  that  specific  organization.  The  standardized 
importance  factor  criteria  are  summarized  below  in  Table  B.3. 

Table  B-3:  Practice  Count  Importance  Factor 


Importance  Factor  Value 

Criteria 

Definition 

1 

0  Codes 

Low  importance 

2 

1  Code 

Important 

3 

2  Codes 

Very  Important 

4 

>2  Codes 

Essential 

Using  the  standardized  importance  factors,  the  organizations  were  separated  into  domains  by  lanes  of  acquisition, 
and  the  total  number  of  citations  for  each  practice  was  standardized.  The  standardized  importance  values  were  then 
averaged  for  each  practice.  Two  of  the  organizations  were  classified  as  both  Lab  Demo/Ops  Prototype  and  Rapid 
Platform  Engineering  based  on  their  activity.  Resultantly,  three  plots  were  produced  to  show  the  average  perceived 
importance  of  each  practice  for  the  organizations  with  each  of  the  three  lanes  of  acquisition:  Rapid  Integrated 
Solutions,  Rapid  Platform  Engineering  and  Lab  demo/Ops  Prototype. 

The  plots  are  shown  in  Figure  B.4.  Note  that  the  lanes  of  acquisition  later  were  iterated  to  4  lanes  (versus  3  as 
shown  in  the  figure),  but  the  ATLAS.ti™  analysis  was  conducted  prior  to  this  point. 
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INTEGRATED  SOLUTION 
AVERAGE  PERCEIVED  IMPORTANCE 


Use  Mature  Tectmology&Roduc* 
Complexity  Focus  on  the  State  of  the 

I'm  .  |M. 


Right  sizing  the  Product. ..Elimaiates 
Major  Program  Oversight 


liur  ci  i  Hi  iU  I  De|>loYinent 
(Development)  is  Part  ot  tire  Product 
Plan 


Government  Team  Leads  the  Way 


Maintain  High  Levels  of  Motivation  and 
Expectations 


Populate  Vom  I earn  with  Specific  Skill \ 
and  Experience 


Build  and  Maintain  Trust 


Keep  anfye  on  Normalization 


PLATFORM  ENGINEERING 
AVERAGE  PERCEIVED  IMPORTANCE 


Use  Mature  Tec hnologyS  Reduce 
Complexity -Focus  on  the  State  of  the 
Possible 


Right  sizing  tire  Product.-.EIinrmates 
Major  Program  Oversight 


Incremental  Deployment 
(Development)  is  Part  of  the  Product 
Plan 


LAB  DEMO  /  OPERATIONAL  PROTOTYPE 
AVERAGE  PERCEIVED  IMPORTANCE 


Use  Mature  Technology'S  Reduce 
Complexity -Focus  on  the  State  of  the 
Possible 


Right  sizing  the  Prodirct...ElitnBtales 
Major  Program  Oversight 


Incremental  Deployment 
(Development)  is  Part  of  the  Product 
Plan 


Government  learn  Leads  the  Way 


Defined  Set  of  Stalile  Rr<|uii entente 
focused  on  Warfighter  Needs 


Maintain  High  Levels  ot  Motivation  and 
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Allowed 


and  Experience 


leslgnlngout  All  Risk  takes 
For  ever. ..Accept  Some  Risk 


Figure  B-4:  Average  Perceived  Importance  (Standardized)  by  Lanes  of  Acquisition 
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For  comparison,  all  three  importance  plots  are  presented  together  to  show  differences  and  similarities  between  the 
lanes  of  acquisition  in  Figure  B-5. 


-♦-INTEGRATED  SOLUTION  LAB  DEMO/ OPERATIONAL  PROTOTYPE  PLATFORM  ENGINEERING 


Use  Mature Technology  & Reduce 
Complexity -Focus  on  the  State  of  the 
Possible 


Maintain  High  Levels  of  Motivation  and 
Expectations 


Work  to  Exploit  Maximum  Flexibility 
Allowed 


Populate  Your  Team  with  Specific 
and  Experience 


Risk  takes 
Forever.. .Accept  Some  Risk 


Right-sizing  the  Product.. .Eliminates 
Major  Program  Oversight 


Incremental  Deployment 
(Development)  is  Part  of  the  Product 
Plan 


GovernmentTeam  Leads  the  Way 


Defined  Set  of  Stable  Requirements 
focused  on  Warfighter  Needs 


Build  and  Maintain  Trust 


an  Eye  on  Normalization 


Figure  B-5:  Average  Perceived  Importance  of  Practices  by  Domain 

Some  of  the  results  shown  above  could  be  expected.  For  example,  organizations  classified  as  Lab  Demo/Operational 
Prototype  show  a  lower  importance  on  the  practice  "Keep  an  Eye  on  'Normalization'"  as  compared  to  those 
organizations  executing  Platform  Engineering  or  integrated  Solutions,  which  should  be  expected.  All  of  the 
organizations  appeared  to  value  practices  relating  to  team  skills  and  experience  and  trust  relationships  within  the 
team,  harkening  back  to  the  previous  conjecture  that  the  right  people  and  team  environment  can  significantly 
contribute  to  the  tenants  that  drive  Rapid.  Finally,  the  perceived  importance  plots  seem  to  indicate  a  fairly  common 
shape,  showing  relative  correlation  between  rapid  organizations  across  three  distinct  lanes  of  acquisition.  One 
exception  is  regarding  the  practice  of  "Maintain  High  Levels  of  Motivation  and  Expectations"  which  was  observed  as 
less  important  in  data  from  organizations  in  the  Integrated  Solution  lane.  While  potential  conclusions  are  presented 
elsewhere,  data  analysis  shows  that  a  set  of  common  practices  did  emerge  among  rapid  organizations,  with 
emphasis  on  people,  requirements,  mature  technology  and  incremental  deployment  and  development  surfacing  as 
commonly  occurring  and  increasingly  important  across  the  spectrum  of  organizations  sampled. 
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Appendix  C:  A  Portfolio  of  SERC  Research  Related  to  Rapid  Development 


C.l  Introduction 

It  became  clear  during  the  early  execution  of  this  Research  Task,  that  many  of  the  other  SERC  RTs  may  generally 
improve  Systems  Engineering  within  any  program,  with  a  side  effect  that  programs  may  be  more  effective,  efficient, 
and  ideally  quicker.  This  Appendix  examines  the  relationships  between  RT-34  and  a  number  of  other  SERC  RTs.  RT- 
34  is  the  first  SERC  RT  to  look  at  the  "research  program"  level  of  other  SERC  research  to  see  how  to  best  leverage 
multiple  investments  into  an  overall  category  of  "expedited  development."  The  data  on  the  other  RTs  was  gathered 
from  publicly  available  descriptions  and  research  reports  posted  on  the  SERC  website  at:  www.SERCuarc.org.  Some 
RTs  may  have  more  accomplishments  than  described  herein,  but  the  point  was  to  take  a  consistent  view  across  what 
was  available.  This  exercise  provides  RT-34  with  additional  support  for  our  findings  (extracted  through  interviews), 
and  also  can  put  other  RT  recommendations,  observations  and  findings  in  context  to  rapid  acquisition  for  urgent 
needs.  Table  C-l  below  shows  the  RTs  examined. 

Table  C-l:  SERC  RTs  Selected  for  Comparison 


SERC  Research  Task 

Description  of  the  Research 

MPT  (Methods,  Processes,  and  Tools) 

Evaluation  of  Systems  Engineering  MPTs 

TO-OOl 

Early  Identification  of  SE  Related  Program  Risks 

RT-10 

Systems  Engineering  Transformation 

RT-18 

Value  of  Flexibility 

RT-19 

Building  Education  and  Workforce  Capacity  in  Systems  Engineering 

RT-24 

Integration  of  Modeling  and  Simulation,  Software  Design,  and  DoDAF 

RT-25 

Requirements  Management  for  Net-Centric  Enterprises 

RT-27 

System  Maturity  and  Architecture  Assessment 

RT-30/31 

Graphical  CONOPS 

RT-35 

Kanban  in  Systems  Engineering 

In  particular,  Graphical  CONOPS,  valuing  flexibility,  and  methods  for  requirements  engineering  were  all  found  to  be 
useful  in  the  concept  and  requirements  development  phase  of  systems  engineering.  The  requirements  engineering 
research  task  specifically  looked  at  large  IT  mergers,  though  some  of  the  principles  may  be  useful  for  overall  systems 
engineering.  Valuing  flexibility  looks  at  ways  to  quantitatively  measure  the  value  of  engineering  flexibility  into 
systems,  which  is  useful  for  trade-off  analysis.  System  Readiness  Levels  looked  at  ways  of  measuring  the  maturity  of 
existing  systems,  which  is  useful  when  using  pre-existing  systems  and  technologies  to  speed  the  development  of  a 
new  system.  Kanban  scheduling  proposed  a  method  of  scheduling  and  management  that  enabled  a  team  to  have 
deliverable  and  testable  products  as  quickly  as  possible.  Integrated  modeling  focused  on  methods  of  picking  the 
appropriate  acquisition  framework  for  a  project,  which  may  help  tailoring  and  speeding  up  acquisition  by  enabling 
teams  to  find  the  quickest  and  most  effective  acquisition  framework  for  their  project.  Workforce  development 
looked  at  ways  to  speed  the  education  of  systems  engineers,  as  well  as  ways  to  spark  student  interest  in  systems 
engineering.  This  could  be  used  to  "build"  systems  engineers  with  the  necessary  skills  and  backgrounds  for  any 
project,  as  well  as  developing  trust  between  engineers.  This  could  also  go  the  other  way,  as  expedited  SE  can  provide 
a  method  for  students  to  see  an  entire  acquisition  process,  where  normally  they  would  only  be  able  to  see  a  small 
milestone  due  to  the  length  of  DoD  acquisition  processes.  Workforce  development  recommendations  are  further 
explored  in  Chapter  5.  Systems  Engineering  Transformation  proposed  eight  areas  of  systems  engineering  that  have 
since  evolved,  many  of  which  are  addressed  in  other  research  topics  that  were  analyzed  for  RT-34. 
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Figure  C-l  puts  the  RT-34  findings  into  perspective  with  the  selected  RTs  that  apply  to  expedited  development.  The 
rest  of  this  appendix  takes  each  RT  examined  and  includes  a  "description"  and  the  "application  to  RT-34." 


Figure  C-l:  SERC  RT  Relationship  to  RT-34  Framework 


C.2  SERC  Research  Task -MPT  (Methods,  Processes,  and  Tools) 


C.2.1  Description:  Evaluation  of  Systems  Engineering  MPTs 

Overview:  The  purpose  of  RT-MPT  is  to  address  SE  shortfalls,  specifically  in  quick  response,  network-enabled  and 
emergent  project  areas.  Phase  I  was  focused  on  developing  useful  MPTs  in  those  areas,  and  identifying  gaps  where 
no  useful  MPTs  could  be  found  for  future  research.  Phase  II  was  focused  on  gathering  further  information  on  those 
MPTs  and  developing  a  taxonomy  for  them,  investigating  micro-process  modeling  techniques  to  support  the 
definition  and  evaluation  of  the  MPTs,  and  to  provide  an  implementation  guidance  on  the  MPTs. 

Phase  I:  For  Phase  I,  the  research  team  conducted  surveys  of  industry  programs,  held  interviews  and  did  a  literature 
search  to  come  up  with  three  MPTs  to  increase  effectiveness  in  the  SE  environments  stated  above. 

1.  Scrum  (lightweight  project  management  methodology,  only  addresses  project  management  and  planning) 

2.  Rapid  Prototyping 

3.  Continuous  Integration 

Three  gaps  were  also  identified  where  no  or  few  useful  MPTs  could  be  found. 

1.  Decision  management 

2.  Stakeholder  requirements  definition 

3.  Sustainment 

Further  research  was  deemed  necessary  in  another  three  areas. 

1.  Mitigating  environmental  constraints 

2.  Refining  definition  of  current  state  of  SE  practice 

3.  Accelerating  MPT  adoption 
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Finally,  four  key  challenges  to  effective  SE  were  observed  and  defined. 

1.  Requirements-  Changing  and  emergent  requirements  and  priorities 

2.  Stakeholders-  Obtaining  useful  stakeholder  input  and  dealing  with  conflicting  stakeholder  requirements 

3.  Sustainment-  Conflicts  between  developing  new  capabilities  and  supporting  current  systems 

4.  Integration-  Integrating  independently  evolving  components  into  a  larger,  interoperable  system 

Phase  II:  Phase  II  had  three  main  focuses. 

1.  Gather  information  on  MPTs  developed  in  Phase  I  and  develop  a  taxonomy  for  them 

2.  Investigate  micro-process  modeling  techniques  to  support  definition  and  evaluation  of  MPTs 

3.  Provide  implementation  guidance  on  MPTs 

Towards  this  end,  follow-up  interviews  were  conducted,  and  research  was  done  to  investigate  the  MPTs  from  Phase  I. 

The  taxonomy  developed  was  based  on  previous  information,  augmented  by  interviews.  It  is  used  to  describe  the 
three  recommended  MPTs,  and  was  shown  to  be  useful  when  validating  and  implementing  the  MPTs. 

Micro-process  modeling  was  used  to  develop  models  of  Scrum.  Analyses  were  successfully  performed  with  models, 
validating  the  use  of  such  modeling  techniques. 

Implementation  guidance  was  developed  using  the  taxonomy,  as  well  as  a  generic  understanding  of  the 
environments  the  MPTs  were  to  be  implemented  in. 


C.2.2  Application  to  RT-34:  RT-MPT 

As  this  research  defined  three  different  MPTs  useful  for  rapid  response  programs,  each  will  be  analyzed  separately. 

Where:  Scrum  is  mainly  applicable  to  requirements  development  under  the  process  aspect  of  level  1  and  the  people 
aspect  of  level  1,  namely  with  respect  to  having  a  team  one  can  trust.  It  can  also  apply  to  the  real-time  management 
aspect  of  level  3.  One  focus  in  Scrum  is  to  determine  the  project  requirements  collaboratively  with  all  stakeholders, 
and  then  break  them  down  into  independent  features.  Also,  the  project  manager  only  addresses  management  and 
planning,  leaving  the  technical  processes  in  the  hands  of  his  team,  which  inherently  involves  a  certain  amount  of 
trust.  This  management  style  can  then  also  be  applied  to  real-time  management  strategies. 

Rapid  prototyping  doesn't  fit  exactly  into  the  expedited  SE  framework.  It  fits  most  closely  to  the  risk-focused  culture 
aspect  of  Level  2,  as  the  purpose  of  rapid  prototyping  is  to  create  a  mock-up  of  a  full  system  to  determine  and 
mitigate  possible  sources  of  risk. 

Continuous  integration  fits  closest  to  the  intense  knowledge  sharing  aspect  of  level  2.  It  allows  complete  project 
visibility  from  everyone  on  the  team,  and  ensures  that  all  parts  work  together  at  every  step  of  the  way,  minimizing 
the  possibility  of  wasted  time  due  to  a  redesign  caused  by  incompatible  parts. 

How:  Scrum  is  a  method  that  should  be  tested  on  smaller  projects  before  applied  to  a  full  scale,  high  priority  task.  As 
far  as  implementing  it,  one  needs  to  find  a  project  manager  that  trusts  his  team  enough  to  let  them  do  the  majority 
of  a  project  without  much  management. 

For  rapid  prototyping,  the  team  must  be  skilled  at  creating  minimal  representations  of  the  system  they're 
developing  and  be  able  to  do  it  accurately  and  in  a  short  time  frame  in  order  to  keep  the  process  expedited.  Also, 
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being  able  to  look  at  a  crude  representation  of  the  full  system  and  see  where  improvements  need  to  be  made  is 
invaluable. 

Though  primarily  for  software  development,  continuous  integration  can  be  utilized  for  mechanical  systems  as  well, 
though  this  may  necessitate  that  integration  is  done  slightly  less  frequently  due  to  the  more  complicated  process  of 
mechanical  integration.  Setting  aside  time  for  integration  is  a  managerial  scheduling  task. 

When:  The  decision  to  use  scrum  needs  to  be  made  well  before  a  project  starts,  and  then  it  must  be  followed 
through  the  entire  lifecycle  of  the  project.  The  trust  inherent  in  this  managerial  system,  however,  needs  to  be 
developed  over  time. 

The  timeframe  for  rapid  prototyping  can  change  from  project  to  project  depending  on  the  amount  of  work  to  be 
done,  but  in  general  it  should  be  started  early  to  determine  any  errors  or  omissions  with  regards  to  the  system 
requirements,  and  to  ensure  that  the  system  is  fully  functional. 

Continuous  integration  can  be  started  as  soon  as  all  team  members  have  some  form  of  deliverable  from  their  work. 
Similar  to  rapid  prototyping,  this  should  be  started  early  and  done  consistently  to  ensure  the  team  is  staying  on  task 
and  is  developing  parts  that  will  work  together  seamlessly  towards  the  fulfillment  of  the  requirements. 


C.3  SERC  Research  Task-TO-OOI 


C.3.1  Description:  Early  Identification  of  SE  Related  Program  Risks 

Overview:  The  DoD  needs  effective  Systems  Engineering  programs  in  order  to  succeed.  These  systems  must  stay  on 
budget  and  on  schedule  in  order  to  be  effective.  As  such,  a  method  must  be  devised  in  order  to  determine  any  risks  to 
achieving  effective  Systems  Engineering  and  ensure  that  the  system  will  meet  budget  and  scheduling  requirements. 

These  risks  are  generally  a  result  of: 

•  Inadequate  program  funding 

•  Misguided  contract  provisions  (not  addressing  key  program  parameter  risks) 

•  Leaving  difficult  tasks  for  later  in  order  to  show  early  progress 

Unaddressed  risks  typically  lead  to  overrunning  budget  and  time  constraints. 

Current  Practices:  Current  methods  for  examining  systems  often  neglect  to  support  evidence  about  how  well  the 
elements  of  a  system  will  perform,  how  expensive  they  will  be,  or  how  compatible  their  underlying  assumptions  are. 
They  tend  to  stick  to  best-case  scenarios  without  examining  if  the  system  could  handle  "rainy-day"  scenarios,  be 
buildable  within  given  budget  and  time  constrains,  or  give  positive  return  on  investments  for  stakeholders. 

Proposed  System:  Through  several  literature  reviews,  Critical  Success  Factors  (CSFs)  were  identified  that  must  be 
met  in  order  for  a  system  to  avoid  risk  and  be  effective.  23  CSFs  were  identified,  relating  both  to  the  performance  of 
the  system  and  the  competency  of  the  people  involved.  These  CSFs  were  organized  into  five  overall  goals: 

•  Concurrent  definition  of  system  requirements  &  solutions 

•  System  life-cycle  organization,  planning  &  staffing 

•  Technology  maturing  &  architecting 

•  Evidence-based  progress  monitoring  &  commitment  reviews 
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•  Professional  and  interpersonal  skills 

In  order  to  determine  if  these  goals  and  CSFs  were  met,  a  series  of  81  questions  was  devised.  For  each  system  that 
was  being  analyzed,  the  questions  were  categorized  based  on  the  relative  impact  a  negative  answer  would  have  on 
the  system,  measured  by  a  percent  overrun  of  budget  and  time  constraints.  They  were  separated  into  critical  (40- 
100%  overrun),  significant  (20-40%),  moderate  (2-20%),  and  little  to  no  impact  (0-2%).  For  each  question,  evidence  is 
provided  in  the  form  of  prototypes,  models,  simulations,  etc.  The  strength  of  the  evidence  is  based  on  the  evaluation 
of  this  evidence  by  impartial  experts.  The  strength  of  the  answer  is  then  used  to  determine  the  risk  probability 
which,  when  multiplied  by  the  risk  impact  gives  the  total  risk  exposure  for  that  area.  Each  CSF  is  then  given  a  risk 
rating  equal  to  the  highest  risk  rating  among  the  questions  relating  to  it. 


C.3.2  Application  to  RT-34:  RT-TO-001 

This  task  order  was  concerned  with  SE  risks  that  cause  projects  to  go  over  the  scheduled  time  or  allotted  budget. 

This  is  different  from  the  risk  in  principle  8  of  the  expedited  systems  engineering  framework,  which  is  primarily  the 
risk  associated  with  a  system  after  it  has  been  deployed. 

Where:  The  research  from  Task  Order  001  relates  most  closely  to  the  risk-focused  culture  aspect  of  level  2.  It  deals 
primarily  with  finding  methods  of  mitigating  risk  associated  with  systems  engineering  practices.  The  methods  used 
to  accomplish  this,  however,  fit  more  into  the  requirements  development  phase  of  a  project,  under  the  process 
aspect  of  level  1,  as  well  as  maintaining  a  small  (but  not  too  small)  budget  which  fits  under  the  product  aspect  of 
level  1. 

How:  Using  the  MPTs  defined  by  this  Task  Order  it  is  possible  to  find  where  the  risk  of  failure  is  in  a  given  project.  If 
people  are  aware  of  where  the  risk  is  and  how  much  of  a  threat  it  is  to  the  project  running  over  time  or  budget,  they 
can  take  measures  to  mitigate  and  eliminate  this  risk  before  it  causes  lasting  problems.  With  success  such  a  large 
issue  with  regards  to  expedited  systems  engineering,  the  ability  to  at  least  ensure  that  a  project  won't  fail  outright  is 
a  step  towards  guaranteeing  the  overall  success  of  the  project. 

When:  The  MPTs  for  identifying  risk  can  be  used  at  any  point  in  a  project,  but  would  be  most  effective  if  used  in  the 
early  planning  stages,  before  budgets  or  schedules  are  set  in  stone,  so  that  they  may  be  changed  in  order  to  mitigate 
risk.  Areas  of  risk  should  then  also  be  monitored  throughout  the  life  cycle  of  the  program  so  that  if  a  problem  arises, 
it  is  recognized  sooner  rather  than  later  and  can  be  dealt  with  before  its  effects  begin  to  inflate.  Several  follow-up, 
total  system  risk  assessments  may  also  be  conducive  to  risk  identification  and  mitigation. 


C.4  SERC  Research  Task  -  RT-10 


C.4.1.  Description:  Systems  Engineering  Transformation 

Overview:  Current  Systems  Engineering  was  developed  primarily  for  the  Space  Race  in  the  1960s  and  1970s  and  is 
deemed  to  be  outdated  by  current  trends  in  technology  and  design  expectations.  Research  Task  10  is  dedicated  to 
creating  a  three  year  road  map  of  research  that  will  transform  the  Methods,  Processes  and  Tools  used  in  Systems 
Engineering  to  be  better  suited  to  current  needs  and  make  it  more  agile,  integrated,  efficient,  leveraged,  and 
deployable. 

Trends:  Several  trends  were  described  in  current  society,  each  of  which  was  incompatible  with  standard  Systems 
Engineering  and  are  as  follows: 
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1.  More  complex  Systems  of  Systems  (SoS)  as  opposed  to  single,  stand-alone  systems.  This  complicates  the 
design  process  as  one  must  be  aware  of  all  social,  economic,  political,  technical,  behavioral,  or 
environmental  issues  involved  in  order  to  develop  an  optimal  system. 

2.  Increased  dependence  on  advanced  yet  unpredictable  SoS  makes  them  a  necessity  rather  than  a 
convenience.  Systems,  services  and  users  are  now  all  expected  to  interact  and  collaborate. 

3.  Increased  customer  expectations  and  competitive  demands  shorten  the  lifecycle  of  services  and  products. 

4.  The  increased  dependency  on  networked  systems  increases  their  value  as  a  target,  and  their  increased 
interconnectivity  increases  their  vulnerability.  This  increases  the  importance  of  security,  especially  when 
dealing  with  SoS. 

5.  Decreased  occurrence  of  "brand-new"  ideas  and  designs.  Systems  must  work  with  more  and  more  legacy 
systems  which  may  not  be  suited  to  their  future  missions  as  a  result  of  this  future  usage  being  unforeseen  at 
the  time  of  their  initial  design. 

6.  The  workforce  is  becoming  more  interactive  and  experiential,  and  more  accustomed  to  change  and  lack  of 
knowledge  of  a  system. 

Domains  of  Transformation:  RT-10  described  eight  domains  that  must  be  researched  in  order  to  transform  Systems 
Engineering. 

1.  Prioritization  &  Tradeoff  Analysis:  Rather  than  creating  a  100%  solution  while  mitigating  all  risk,  a  tradeoff 
must  be  made  between  budget,  usability,  security,  overall  value,  and  time  to  completion.  Techniques  must 
be  researched  to  analyze  prioritization  tradeoffs  to  improve  quality  of  decision  making  and  reduce  time  to 
completion. 

2.  Concept  Engineering:  Interactive,  multimedia  environments  must  be  developed  to  quickly  create  CONOPS 
and  models  in  conjunction  with  multiple  stakeholders  to  speed  the  design  process  and  allow  all  involved  to 
get  an  idea  of  what  the  system  does,  as  well  as  its  value. 

3.  Architecture  and  Design  Analysis:  Visualization  and  analysis  tools  must  be  developed  to  facilitate  decision 
making  through  specification  and  design  processes.  Such  tools  could  help  determine  how  and  where  a 
system  should  be  modified  while  minimizing  the  impact  on  design  and  testing. 

4.  Design  &  Test  Reuse  and  Synthesis:  A  method  is  needed  to  categorize  and  catalog  existing  designs  and 
technology  for  reuse.  Methods  to  synthesize  existing  architectures  for  use  in  new  designs  using  these 
databases  must  be  explored. 

5.  Active  System  Characterization:  Methods  and  tools  must  be  developed  to  facilitate  real-time  analysis  of 
existing  systems  to  determine  their  value  as  well  as  their  usability  in  future  designs. 

6.  Human-System  Integration:  Models  are  needed  to  determine  how  humans  will  interact  with  the  system,  as 
well  as  the  capabilities  and  limitations  of  the  people  designing  the  system. 

7.  Agile  Process  Engineering:  Processes  must  be  created  to  rapidly  develop  systems  that  are  agile  and 
adaptable  to  changes  in  tools  and  technology,  in  environments  that  may  not  be  suited  to  agile  development. 

8.  Modeling  Environment  Infrastructure:  There  must  be  a  way  to  provide  effective  communication  between 
the  created  models  and  the  users  of  those  models,  and  the  design  environments. 


C.4.2  Application  to  RT-34:  RT-10 

Due  to  the  fact  that  the  purpose  of  RT-10  was  to  create  a  roadmap  for  future  research,  analysis  will  be  done  on  each 
of  the  individual  domains  stated  above.  The  relationship  to  RT-34  of  these  domains  when  the  proposed  research  on 
them  is  complete  will  be  analyzed.  Each  domain  will  be  referred  to  by  its  number  in  the  above  summary  for 
convenience. 

Where:  Domain  6  is  the  only  domain  related  to  the  people  aspect  in  level  1  of  the  expedited  SE  framework.  Being 
able  to  determine  the  capabilities  and  limitations  of  the  people  developing  a  system  could  prove  useful  for  picking  a 
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team  with  the  right  skills  and  motivation,  and  then  being  able  to  trust  them  with  working  on  an  expedited  project. 
This  could  also  help  in  the  Organization  Ambidexterity  aspect  of  level  2,  as  it  will  enable  picking  people  capable  of 
working  under  several  disciplines. 

Domain  1  fits  into  the  Process  aspect  of  level  1,  as  it  works  under  the  assumption  that  some  risk  is  acceptable,  and 
works  to  create  a  trade-off  between  risk,  cost,  schedule,  etc.,  in  order  to  build  a  design  a  quality  product  rapidly.  It  is 
also  integrated  into  the  Risk-Focused  Culture  aspect  of  level  2,  as  having  people  that  are  accustomed  to  risk  and  are 
comfortable  dealing  with  and  mitigating  it  will  be  able  to  use  this  tool  more  effectively  to  speed  up  a  project. 

Domain  2  also  fits  into  the  process  aspect  of  level  1.  Having  a  method  to  quickly  and  reliably  come  up  with  the 
requirements  of  a  system,  enables  a  team  to  start  work  on  the  system  faster,  and  make  design  changes  due  to 
unclear  requirements  less  likely 

Domain  3  is  most  relevant  in  the  product  aspect  of  level  1,  as  it  deals  mostly  with  mature  technology  and  developing 
methods  of  finding  existing  technologies  relevant  to  current  design  problems. 

Domain  7  relates  most  closely  to  the  Flexible  Acquisition  Practices  of  level  3.  Being  able  to  tailor  acquisition 
processes,  especially  in  big  acquisition,  can  drastically  speed  up  a  project. 

Domain  8  fits  closest  to  the  process  aspect  of  level  1,  specifically  with  regard  to  requirements.  It  ensures  that  the 
time  spend  developing  detailed  requirements  to  the  satisfaction  of  the  user  isn't  put  to  waste  by  veering  away  from 
the  proposed  design. 

Domains  4  and  5  help  to  facilitate  the  use  of  mature  technology  for  the  product  aspect  of  level  1.  These  domains 
play  off  each  other,  as  domain  4  seeks  to  create  a  database  of  existing  technologies,  enabling  engineers  to  search  for 
previously  designed  components  that  may  solve  problems  they  are  facing,  and  domain  5  seeks  to  categorize  existing 
systems  to  fit  into  the  database  created  by  domain  4,  as  well  as  defining  their  capabilities  and  behaviors  to  ensure 
that  they  are  utilized  correctly  in  future  systems. 

How:  Domain  1  can  be  utilized  to  determine  exactly  what  will  be  delivered  at  the  end  of  the  project.  After 
conversations  with  stakeholders  to  determine  how  much  risk  they're  willing  to  accept,  the  time  frame  they  need  a 
product  in,  and  their  budget,  the  result  of  research  in  this  area  will  be  used  to  quickly  identify  just  how  much  of  the 
total  solution  can  be  reasonably  designed  with  those  given  resources,  and  how  much  value  it  will  give  to  the 
stakeholders.  Based  on  that,  requirements  can  be  modified  until  all  stakeholders  are  satisfied  with  the  end  result. 

Domain  2  is  useful  for  setting  the  requirements  for  a  project,  which  can  be  very  conducive  for  rapid  development 
and  deployment.  Specifically,  the  fact  that  all  stakeholders  are  able  to  participate  in  the  concept  engineering  will 
negate  the  possibility  of  designing  a  product  or  system  that  doesn't  fit  the  stakeholder's  needs,  as  they  will  be 
involved  with  every  step  of  development  to  ensure  they  are  getting  what  it  is  they  need. 

Domain  3  takes  some  of  the  risk  out  of  the  design  process  by  helping  engineers  see  what  parts  of  a  system  they  can 
modify  with  minimal  impact  on  the  design.  This  in  turn  doesn't  necessitate  as  much  testing  as  if  a  modification 
would  have  an  unknown  or  large  impact  on  a  system.  This  can  save  the  time  that  would  otherwise  be  spent  on  extra 
testing,  as  well  as  mitigating  some  of  the  risk  associated  with  modifying  systems. 

Domain  4  can  inform  engineers  what  legacy  systems  and  mature  technologies  are  available  much  easier  and  faster 
than  may  otherwise  be  possible.  Also,  if  existing  systems  and  products  are  made  more  accessible,  one  may  be  found 
to  fit  a  stakeholder's  requirements  that  may  not  have  been  found  otherwise,  saving  a  substantial  amount  of  time 
that  would  have  been  required  to  design  a  system  or  part  that  already  exists. 
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Domain  5  offers  a  method  to  provide  feedback  between  the  virtual  and  physical  domains  of  a  system.  The  up  to  date 
data  it  provides  allows  real-time  updates  of  models  of  the  technical  systems  and  of  human  behavior  and  system 
usage  of  the  system  being  developed,  as  well  as  the  system  doing  the  developing.  These  updates  can  be  used  to 
determine  the  effects  of  changes  made  to  the  system  in  terms  of  time,  effort,  cost  and  risk,  allowing  engineers  and 
designers  to  see  the  consequences  of  any  modifications  they  may  be  considering  before  they  are  made,  possibly 
saving  precious  time  and  resources. 

Domain  6  allows  optimization  with  regards  to  the  human  aspect  of  a  system.  This  domain  can  be  used  to  model 
human  interactions  with  the  developed  system  to  ensure  that  it  responds  in  a  way  suited  to  fulfilling  the  system's 
requirements.  This  makes  it  possible  to  know  that  a  developed  system  will  be  fully  useable  by  the  warfighter  it  was 
developed  for. 

Domain  7  provides  the  processes  and  governance  to  enable  parallel  development  in  the  aforementioned  domains. 
The  processes  developed  in  this  domain  function  primarily  to  facilitate  the  interactions  between  the  other  domains, 
ensuring  that  they  all  work  cohesively  to  enable  rapid  and  agile  development.  This  domain  paired  with  domain  8 
make  up  the  environment,  processes  and  governance  that  the  other  domains  will  work  within. 

Domain  8  provides  an  infrastructure  to  support  all  the  aforementioned  domains,  as  well  as  providing  an  effective 
interface  to  users  of  the  system.  This  provides  a  means  to  realize  the  agile  processes  in  operation.  This  enables 
systems  engineering  to  be  an  integrated,  embedded  capability  in  the  support  of  systems  throughout  their  lifecycle. 

When:  Domains  1  through  3  are  all  primarily  concerned  with  developing  requirements  and  picking  an  effective 
design.  As  a  result,  these  three  modules  will  be  implemented  very  early  in  the  systems  engineering  process.  Starting 
with  domain  1  to  create  a  set  of  requirements,  these  requirements  can  then  be  used  alongside  domain  2  in  order  to 
develop  a  design  with  models  and  a  full  set  of  CONOPS  that  satisfies  the  stakeholders.  This  can  then  be  used  in 
conjunction  with  domain  3  in  order  to  pick  the  optimal  design  based  on  analysis  done  through  domain  1. 

Domain  6  can  be  implemented  in  the  earlier  phases  of  design  and  used  throughout  the  majority  of  the  design 
process.  Creating  models  that  take  human  interactions  with  the  system  into  consideration  is  important  when 
choosing  a  design,  and  can  then  be  utilized  during  the  entire  design  and  development  process  in  order  to  ensure 
that  the  system  being  designed  is  optimized  in  its  capabilities  both  from  a  technical  and  human  aspect. 

Domain  5  is  relevant  in  several  different  systems  design  and  engineering  phases,  each  in  unique  ways.  In  the  earlier 
phases  of  systems  design,  existing  systems  that  have  been  analyzed  and  characterized  using  the  MPTs  of  domain  5 
can  be  looked  at  for  possible  uses  in  new  development.  In  later  phases,  after  a  system  has  been  developed  and  is 
being  deployed,  this  system  can  be  characterized  for  possible  use  in  future  systems. 

Domain  4,  when  the  proposed  system  has  been  completed,  can  also  be  used  early  in  the  design  process  to  find 
existing  systems  that  may  be  implementable  with  a  current  systems  engineering  project. 

As  a  result  of  domains  7  and  8  primarily  dealing  with  the  integration  of  the  above  domains  and  the  creation  of 
interfaces,  environments  and  architectures  with  which  to  utilize  these  MPTs,  these  two  remaining  domains  are 
utilizable  throughout  the  entire  lifecycle  of  a  system,  from  concept  engineering  and  requirement  analysis,  through 
its  deployment  stage. 
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C.5  SERC  Research  Task  -  RT-18 


C.5.1  Description:  Value  of  Flexibility 

Overview:  Though  it  is  widely  accepted  that  flexibility  is  a  good  thing,  there  is  yet  a  way  to  measure  the  value  of 
flexibility,  especially  with  respect  to  Systems  Engineering.  This  research  task  is  an  effort  to  put  a  quantitative  value 
on  engineering  flexibility  into  systems  in  various  situations,  allowing  DoD  acquisition  personnel  to  make  informed 
decisions  on  "how  much  to  invest  in  flexibility." 

Definition:  The  first  task  is  to  set  a  definition  for  flexibility.  Multiple  definitions  for  flexibility,  as  well  as  definitions  for 
multiple  different  kinds  of  flexibility  were  analyzed  and  five  categories  were  extrapolated  that  were  useful  in 
defining  flexibility: 

•  Whether  or  not  the  system  changes 

•  How  efficient  the  change  is  with  respect  to  time  and  money 

•  The  source  of  the  change  with  respect  to  the  system  (internal,  external,  requirements  based) 

•  Whether  the  change  was  foreseeable  or  not 

•  Whether  the  change  happened  before  or  after  the  system  was  fielded 

Cost:  In  determining  the  value  of  flexibility,  the  first  step  was  to  determine  the  cost  and  what  were  (if  any)  the 
tradeoffs  of  engineering  flexibility  into  the  system.  It  was  speculated  that  adding  flexibility  would  result  in  a  higher 
expenditure  of  resources  and/or  decreases  in  productivity  of  the  final  system.  Flexibility  was  also  shown  to  have 
rapidly  diminishing  marginal  returns  on  investment;  there  was  not  much  more  value  in  a  lot  of  flexibility  than  there 
was  in  a  little  flexibility. 

Benefit:  Through  analysis  of  software  product  line  reuse,  engineering  products  that  will  be  adequate  for  reuse  cost 
50%  more  in  development  and  certification,  but  then  reusing  those  products  without  modification  cost  20%  of  the 
cost  of  redeveloping  components.  These  parts  were  also  shown  to  be  useful  in  lines  of  at  least  three  similar 
products.  If  the  components  were  modified  before  reuse,  20%-50%  was  added  to  the  cost  of  reuse. 

Methods:  In  mechanical  systems,  the  overall  function  is  achieved  via  the  interactions  of  several  sub  systems  and 
components.  This  makes  alterations  to  one  component  difficult  as  any  changes  in  that  may  make  it  incompatible 
with  the  other  components,  and  as  a  result,  when  engineering  with  flexibility  in  mind,  modular  designs  should  be 
chosen  so  that  alterations  to  one  subsystem  will  have  less  of  an  impact  on  the  subsystems  it  interacts  with.  Initially, 
any  foreseen  changes  must  be  integrated  into  the  design,  adding  versatility  and  possibility  for  upgrades  and 
customization.  After  that,  design  must  begin  to  compensate  for  unforeseen  changes. 

Value:  Several  methods  to  quantitatively  determine  the  value  of  flexibility  in  design  were  examined: 

•  Total  Ownership  Cost  (TOC):  The  cost  to  research,  develop,  acquire,  own,  operate,  and  dispose  of  a  system.  This 
takes  into  account  the  extra  costs  associated  with  designing  a  flexible  system,  as  well  as  the  savings  downstream 
in  reusing  the  design.  Creating  a  TOC  of  the  same  system  both  with  and  without  flexibility  allows  one  to  put  a 
monetary  value  on  the  flexibility  and  provides  a  way  to  determine  appropriate  levels  of  investment  in  flexibility. 

•  Hedge  Framework:  Takes  into  account  the  value  of  a  system's  current  capabilities  as  well  as  its  opportunities.  In 
order  to  do  this,  the  possible  opportunities  and  the  possible  environments  that  a  system  may  have  value  in  must 
be  explicit  during  design  and  engineering.  The  approach  is  to  make  models  of  where  and  when  such  a  system 
could  be  valuable.  This  method  is  still  under  development  by  The  University  of  Virginia. 
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•  Knowledge  Value  Added  +  Integrated  Risk  Management  and  Real  Options:  Measures  operating  performance, 
cost  effectiveness,  return  on  investment,  risk,  real  options,  and  analytical  portfolio  optimization.  This  determines 
the  value  based  on  the  return  on  knowledge  and  the  return  on  investment  of  a  system,  quantifying  the  value  of 
processes,  functions,  departments,  divisions,  and  organizations  in  common  units.  It  is  able  to  highlight 
efficiencies  and  inefficiencies  and  estimate  the  total  value  created  by  adding  flexibility. 

Due  to  the  fact  that  RT-18  has  to  date  only  completed  phase  1  of  its  proposed  plan,  phase  1  is  all  that  will  be 
analyzed.  This  phase  specifically  focuses  on  finding  MPTs  that  enable  one  to  place  a  numerical  value  on  engineering 
flexibility  into  a  system. 


C.5.2  Application  toRT-34:  RT-18 

Where:  The  results  of  the  phase  1  research  of  RT-18  is  relevant  in  the  process  aspect  of  level  2,  specifically  with 
regard  to  requirements  and  risk  accepting,  as  well  as  in  the  risk-focused  culture  aspect  of  level  3.  Having  a  numerical 
value  for  designing  flexibility  into  a  system  provides  another  metric  for  trade-off  analysis  between  flexibility,  risk, 
cost,  etc. 

How:  In  order  to  first  put  a  number  on  flexibility  in  a  system,  one  must  use  one  of  the  MPTs  defined  by  phase  1  of 
RT-18.  Then,  using  a  tradeoff  analysis,  possibly  the  one  from  RT-10,  it  must  be  determined  just  how  much  flexibility 
to  engineer  into  a  system  in  collaboration  with  the  stakeholders.  This  must  take  into  account  the  value  as  well  as  the 
cost  of  flexibility  with  regards  to  the  particular  system  being  looked  at.  Perhaps  for  the  system  in  question,  it's  vastly 
more  important  to  have  a  completed  product  on  time,  or  within  budget,  or  with  more  productivity  than  it  is  to  have 
flexibility  engineered  in. 

When:  At  this  phase  of  the  research,  as  it  is  solely  concerned  with  placing  a  value  on  flexibility,  the  timeframe  for  its 
use  is  limited  to  the  early  development  stages,  namely  the  creation  of  requirements  and  capabilities  and  perhaps  the 
CONOPS  development.  The  eventual  end  product  of  MPTs  that  enable  flexibility  to  be  engineered  into  a  system  will 
be  applicable  throughout  the  entire  design  and  development  cycle  of  a  system,  and  possibly  into  its  deployment  and 
reuse. 


C.6  SERC  Research  Task  -  RT-19 


C.6.1  Description:  Building  Education  and  Workforce  Capacity  in  Systems  Engineering 

Overview:  The  goal  of  RT-19  is  to  understand  the  impact  of  a  set  of  diverse  capstone  courses  on  student  learning  of 
and  career  interest  in  Systems  Engineering.  Participating  students  were  exposed  to  real  DoD  problems,  along  with 
differing  course  designs,  structures,  materials,  and  involvement  of  DoD  mentors  to  assess  how  these  differences 
affect  the  student's  learning  and  interest  in  Systems  Engineering. 

Methods:  The  Capstone  project  was  piloted  in  eight  civilian  and  six  military  universities.  Participating  students  from 
these  universities  were  split  into  groups  and  given  one  of  four  real  DoD  problems  to  work  on  over  the  course  of  the 
2010-11  academic  year.  Teams  were  organized  in  various  ways,  the  most  common  being  multiple  teams  working  on 
multiple  different  design  problems. 
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Most  institutions  had  systems  engineering  faculty  lead  and  assist  with  the  development  of  the  course,  though  many 
also  had  faculty  from  various  other  backgrounds  assist  as  well.  11  of  the  14  institutions  had  faculty  from  at  least 
three  different  backgrounds  involved  with  the  project.  Only  one  institution  had  one  faculty  member  involved. 

In  order  to  analyze  the  impact  of  the  capstone  project  on  the  student's  learning  of  SE,  student  interest  in  SE  careers, 
and  student  awareness/interest  in  authentic  DoD  problems,  pre/post  surveys  and  case  study  analyses  were 
collected,  and  student  blogs  were  analyzed,  as  well  as  custom  assessments,  designed  by  the  participating  faculty.  A 
case  study  of  a  Bradley  Fighting  Vehicle  was  done  by  students  both  before  and  after  their  capstone  experiences. 
Using  an  analytic  rubric,  the  complexity  of  the  student's  thinking  using  systems  engineering  was  measured. 

Findings:  After  the  completion  of  the  capstone  courses,  students  were  found  to  be  more  aware  of  specific  problems 
faced  by  the  DoD.  Student's  awareness  was  most  increased  with  respect  energy-related  problems,  and  most 
decreased  with  respect  to  weapons  systems.  Students  were  also  seen  to  use  SE  terminology  significantly  more. 

A  survey  on  student  interest  in  systems  engineering  careers  did  not  show  statistically  significant  increases  in  interest 
in  careers  in  general,  or  in  government  and  industry.  However,  for  students  with  no  prior  SE  experience,  increases  in 
interest  were  shown  in  all  areas,  with  the  highest  average  increase  being  in  industry.  Students  with  prior  SE 
experience  showed  an  increase  in  interest  in  SE  careers  in  general  and  in  industry,  but  decreases  in  government 
careers.  80%  of  students  said  they  would  consider  a  career  in  systems  engineering,  but  that  they  would  prefer  to 
gain  experience  in  their  chosen  field  first.  The  students  that  said  they  wouldn't  consider  a  career  in  systems 
engineering  mainly  cited  their  current  engineering  field  as  the  reason  they  wouldn't. 

The  DoD  mentors  were  seen  to  be  most  beneficial  when  communication  between  mentors  and  students  was 
frequent  and  specific,  such  as  for  design  reviews.  Students  said  the  mentors  helped  them  avoid  "exploring  too  many 
blind  alleys."  In  all  but  one  institution,  however,  students  rated  the  mentor's  usefulness  in  learning  about  and 
applying  SE  in  their  courses  in  the  low-  to  mid-range,  as  compared  with  other  aspects  of  the  capstone  courses. 


C.6.2  Application  to  RT-34:  RT-19 

Where:  RT-19  relates  very  strongly  to  the  people  aspect  of  level  1.  Being  able  to  "grow"  an  engineering  team  with 
the  right  skills,  motivation  and  culture,  and  the  ability  to  trust  them  is  an  enormous  asset  in  expedited  engineering.  It 
could  also  be  beneficial  in  the  organization  ambidexterity  aspect  of  level  3,  as  it  would  be  possible  to  train  engineers 
in  multiple  areas  to  broaden  their  expertise,  as  well  as  making  them  more  comfortable  working  in  areas  other  than 
what  they  specify  in. 

How:  Through  the  MPTs  formed  through  RT-19,  it  may  be  possible  to  pick  students  from  different  backgrounds,  with 
different  experiences  and  skill  sets,  and  form  them  into  systems  engineers  with  a  desired  set  of  skills  or  a  particular 
desired  background.  If  the  students  are  picked  carefully  or  the  courses  are  constructed  in  a  specific  way,  it  may  be 
possible  to  custom  order  and  grow  an  engineer  with  exactly  the  right  background,  skills,  experiences,  etc.,  that  one 
needs  for  a  particular  task. 

When:  As  this  research  task  was  concerned  with  the  impact  of  different  teaching  methods  for  systems  engineering 
on  students,  all  of  it  must  be  done  well  before  any  systems  engineering  design  tasks  are  started.  As  such,  a  modicum 
of  foresight  is  required  with  regards  to  what  problems  may  arise  when  the  next  "batch"  of  systems  engineers  has 
finished  their  courses  and  matured.  This,  an  incredibly  varied  workforce  will  need  to  be  trained  in  order  to  have  the 
right  person  for  the  job,  no  matter  what  that  job  may  be. 
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C.7  SERC  Research  Task  -  RT-24 


C.7.1  Description:  Integration  of  Modeling  and  Simulation,  Software  Design,  and  DoDAF 

Overview:  There  is  little  information  transfer  or  model  reuse  among  architecture  compliance,  simulation,  and 
software  engineering  models,  generally  resulting  in  unnecessary  redundancy,  models  that  are  out  of  sync,  and  a  loss 
of  domain  knowledge  and  information.  This  is  a  direct  result  of  differences  in  methods,  tools  and  data  formats 
between  the  three  areas.  This  Research  Task  investigates  ways  to  integrate  these  areas. 

Different  techniques  for  specification  of  software  requirements  (SysML),  simulation  models  (Arena)  and  enterprise 
architecture  (BPMN)  were  compared  and  the  different  concepts  of  each  language  were  related  to  each  other  using 
DoDAF  2.0  as  a  framework.  A  prototype  converter  between  BPMN  2.0  XML  and  Arena  was  developed  that  allowed 
users  to  reuse  BPMN  models  for  event  simulation. 

Motivation:  Requirements  management  for  software-intensive  systems  are  typically  disconnected  from  the  actual 
codebases  and  architectural  tools  used  to  create  the  models  and  representations  of  the  system  architecture,  as  well 
as  the  simulation  tools  used  to  evaluate  the  architecture.  As  a  result,  developers  and  architects  are  required  to 
manually  link  the  requirements  to  the  data  in  other  tools,  and  need  to  do  this  repeatedly  as  significant  changes 
occur  in  either  environment. 

Project  Phases:  Project  proceeded  in  three  stages: 

1.  Focus  on  methods  and  techniques  for  system  and  software  modeling;  created  mapping  between  different 
modeling  languages 

2.  Focus  on  modeling  views  needed  for  system  documentation  and  useful  for  system  design;  determine 
usefulness  of  DoDAF  for  software  engineering;  map  different  model  types  over  DoDAF  2.0;  determine  extent 
formal  models  cover  DoDAF  2.0 

3.  Prototype  findings  from  (1)  and  (2);  develop  model  convertor;  Transfer  requirements  models  from  news  use 
case  to  executable  workflow  model 

Findings:  A  set  of  recommendations  were  developed  through  the  course  of  the  project: 

1.  Clearly  define  the  key  vocabulary,  including  capabilities,  activities,  resources  and  performers 

a.  Capabilities-  desired  effects  of  system 

b.  Activity-  action  that  transforms  input  into  desired  output 

c.  Resources-  inputs/outputs  to  activities 

d.  Performers-  actors  that  execute  activities 

2.  Understand  that  not  all  perspectives  of  DoDAF  2.0  framework  are  useful  for  software  based  projects.  Useful 
perspectives  include  CV-6,  DIV-1,  OV-6c,  and  SV-lOa 

3.  Process  models  should  be  developed  early  in  order  to  check  completeness  of  architectural  scope  and 
dynamic  behavior  of  intended  system 

4.  Simulation  models  should  be  integrated  into  the  software  design  process,  as  they  allow  developers  to 
estimate  system  performance  and  are  affordable  to  integrate 

Proper  use  of  these  recommendations  helped  facilitate  the  transition  from  requirements  engineering  into  simulation 
and  implementation. 


Contract  Number:  H98230-08-D-0171 


Page  123 

Report  No.  SERC- 2012-TR- 034 


TO  0015,  RT034 


UNCLASSIFIED 


C.7.2  Application  toRT-34:  RT-24 

As  much  of  the  RT-34  research  was  done  on  physical  systems  engineering,  RT-24  could  help  transition  into  software 
systems  development  by  providing  the  basis  of  an  expedited  framework. 

Where:  The  recommendations  concerned  with  modeling  as  well  as,  to  a  lesser  extent,  the  recommendation  to 
define  key  vocabulary  relate  to  the  process  aspect  of  level  1  in  the  expedited  framework,  specifically  with  respect  to 
the  requirements  portion  of  systems  engineering.  Recommendations  as  to  what  acquisition  frameworks  to  use  in 
particular  instances  could  be  useful  in  the  flexible  acquisition  practices  aspect  of  level  3  to  ensure  that  only  useful 
acquisition  techniques  are  being  used. 

How:  The  implementation  of  these  recommendations  starts  at  the  project  manager  level.  He  should  ensure  that  the 
key  vocabulary  is  defined  and  the  useful  DoDAF  frameworks  are  chosen  for  integration  with  the  project.  The 
development  of  models  is  more  so  a  function  of  the  design  team  than  of  the  manager,  though  he  should  be  aware  of 
the  process  and  its  value.  Development  of  these  models  should  begin  shortly  after  the  requirements  are  set— if  not 
sooner— to  ensure  maximum  benefit  is  seen  from  them. 

When:  The  recommendations  would  be  most  useful  in  the  early  stages  of  a  project.  The  first  two  should  be  among 
the  first  things  done  as  they  set  the  foundation  for  the  project  to  come,  and  are  important  to  planning  out  the  stages 
and  methods  of  the  development  of  the  project.  The  third  recommendation  explicitly  states  it  should  be  completed 
early,  specifically  in  parallel  with  or  shortly  after  the  requirements  development.  The  same  goes  for  the  fourth 
recommendation,  and  all  of  these  recommendations  should  be  repeated  with  any  significant  changes  to  the  project. 


C.8  SERC  Research  Task  -  RT-25 


C.8.1  Description:  Requirements  Management  for  Net-Centric  Enterprises 

Overview:  A  Net-Centric  Enterprise  is  a  number  of  semi-autonomous  organizations  that  collaborate  within  the 
context  of  a  federated  structure.  The  goal  of  RT-25  is  to  devise  a  framework  and  a  series  of  MPTs  for  Net-Centric 
Enterprises  to  manage  the  capabilities  and  requirements  of  the  various  organizations  involved  in  the  Net-Centric 
Enterprise,  taking  into  consideration  the  necessity  of  these  units  to  collaborate  using  a  similar  IT  system,  possibly 
requiring  integration  or  merging;  the  existence  of  legacy  systems;  and  the  evolution  of  missions  and  needs  over 
time. 

Methodology:  The  overall  method  devised  by  this  RT  was  to  decompose  the  capabilities  decided  upon  by  the 
participating  units  into  the  requirements  and  architectures  of  the  system,  while  maintaining  traceability  of  these 
requirements  and  architectures  back  to  the  capabilities  they  were  extrapolated  from.  A  three-step  decision  making 
process  was  created  to  facilitate  this  evolution  of  capabilities: 

1.  Decision  Inputs 

a.  Determine  decision  drivers* 1 2  based  on  capabilities,  requirements,  and  architectures  of  system 

b.  Existing  MPTs  (such  as  Component-Bus-System-Property)  exist  to  refine  capabilities 

c.  Capabilities,  requirements  and  architectures  should  be  determined  by  negotiations  between 
participating  stakeholders 

2.  Decision  Process 

a.  Use  decision  drivers  to  determine  cost/risk  and  value  of  capabilities,  requirements  and  architectures 
qualitatively  and  relatively  to  each  other 
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3.  Decision  Priorities 

a.  Use  value  and  risk/cost  to  prioritize  decisions  using  prioritization  plane 
Algorithmic  description  of  using  the  methodology: 

1.  Initialization 

a.  Articulate  top-level  capability  intents 

b.  Identify  net-centric  actors  at  all  levels 

c.  Identify  existing  systems  and  architectures 

d.  Reconcile  capability  intents  into  capabilities 

2.  Iteration  in  Decision  Framework 

a.  Decompose  capabilities  into  functions  into  requirements 

b.  Reconcile  capabilities,  functions,  requirements  and  architectures 

c.  Map  capabilities,  functions  and  requirements  to  architectures 

d.  Incorporate  evolving  needs  over  time 

3.  Progress  Reporting 

a.  Traceability  on  progress  of  top-level  capabilities 

b.  Volatility  and  conflict  assessment 

c.  Re-prioritization,  re-budgeting,  re-scheduling 


C.8.2  Application  to  RT-34:  RT-25 

As  this  Research  Task  is  mainly  concerned  with  management  of  multiple  large  organizations,  it  is  more  suited  to 
large  systems  engineering  tasks,  rather  than  small,  urgent  needs. 

Where:  The  main  focus  of  this  research  was  on  the  development  of  requirements  between  many  large  stakeholders, 
and  primarily  relates  to  the  process  aspect  of  level  2  under  requirements.  It  was  specifically  focused  on  being  able  to 
construct  a  set  of  requirements  from  a  group  of  capabilities,  while  maintaining  traceability  of  the  requirements  back 
to  the  capabilities  they  evolved  from. 

How:  The  MPTs  defined  in  this  RT  were  looked  at  specific  to  collaborations  between  multiple  large  groups  working 
towards  one  large  goal,  each  having  different  capabilities  and  requirements.  The  research  was  done  looking  at  IT 
mergers,  though  the  MTPs  should  be  effective  for  physical  systems.  Several  different  methods  were  cited  to  pick  one 
set  of  capabilities  among  multiple  stakeholders.  Ways  were  then  formulated  to  decompose  those  into  requirements, 
functions  and  architectures  while  at  the  same  time  maintaining  traceability  of  those  lower  functions  back  to  the 
higher  capabilities. 

When:  Like  other  MPTs  concerned  with  developing  requirements  of  systems,  these  must  be  implemented  early  in 
the  engineering  process  in  order  to  be  effective.  Especially  with  multiple  large  stakeholders  and  a  large  project, 
having  a  defined  set  of  requirements  that  everyone  involved  is  aware  of  is  important  to  effective  systems 
engineering.  Additionally,  these  methods  were  designed  for  collaborations  between  large  organizations  and  may  not 
be  suited  to  smaller  projects;  it  might  have  to  wait  until  all  other  areas  of  expedited  systems  engineering  have  been 
scaled  up  to  larger  venues. 
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C.9  SERC  Research  Task  -  RT27 


C.9.1  Description:  System  Maturity  and  Architecture  Assessment  MPTs 

Overview:  The  three  objectives  of  this  research  task  were  to: 

1.  Identify  systems  engineering  and  architectural  artifacts  to  support  the  assessment  of  technology,  integration 
and  system  maturity 

2.  Correlate  systems  engineering  architectural  artifacts  to  supported  views  and  artifacts  in  Department  of 
Defense  Architecture  Framework  (DoDAF),  that  enable  Technology  and  Integration  Readiness  Level 
assessment 

3.  Develop  a  maturity  assessment  tool  that  works  with  standard  industry  SE  architecture  tools 

Background:  Metrics:  measure  the  attributes  of  an  object  of  interest,  allowing  decision  makers  to  make  more 
informed  decisions  regarding  that  object.  Four  rules  to  successful  metrics: 

1.  The  way  value  is  used  should  be  clear 

2.  The  data  to  be  collected  for  the  metric  should  be  easily  understood  and  collected 

3.  The  way  of  deriving  value  from  the  data  should  be  as  simple  and  clean  as  possible 

4.  Those  for  whom  the  use  of  the  metric  implies  additional  cost  should  see  maximum  direct  benefit 

Descriptive  metrics  can  be  objectively  measured,  are  quantifiable,  and  have  minimum  variability  between  observers. 
Prescriptive  metrics  are  qualitative,  judgmental,  subjective,  and  based  on  perceptual  data. 

Strive  to  make  prescriptive  metrics  of  Technology  Readiness  Level  (TRL)  and  Integration  Readiness  Level  (IRL)  more 
descriptive. 

Readiness  Levels:  TRL  has  been  shown  to  be  an  extremely  beneficial  metric  for  assessing  the  risks  associated  with 
developing/acquiring  a  technology;  however  critics  argue  that  TRL  combines  many  dimensions  of  technology 
readiness  into  one  metric.  System  Readiness  Level  (SRL)  was  developed  to  be  a  robust,  repeatable,  agile  assessment 
that  could  easily  be  transferred  to  different  applications  and  architectures.  SRL  was  designed  to  incorporate  both 
TRL  and  IRL,  and  defines  a  series  of  levels  that  articulate  maturation  milestones  for  integration  activities. 

System  Architecture  and  DoDAF:  System  architecture  is  an  arrangement  of  elements  and  subsystems,  and  the 
allocation  of  functions  to  them  to  meet  system  requirements. 

Architectures  help  system  engineers  examine  a  system  from  multiple  perspectives,  and  help  decision  makers  to 
reason  about  a  problem. 

Results:  In  order  to  satisfy  objectives  (1)  and  (2),  the  readiness  levels  were  mapped  to  DoDAF.  The  first  step  was  to 
use  the  TRL  Calculator  and  IRL  decision  criteria  to  define  the  decision  criteria  for  both.  The  next  step  was  to  decide 
which  DoDAF  Conceptual  Data  Models  could  best  address  each  readiness  level  decision  criteria.  The  finals  steps 
were  to  select  models  from  a  subset  list  of  all  DoDAF  models. 

In  order  to  satisfy  objective  (3),  an  SRL  calculator  was  developed  that  extracted  information  from  the  TRL  and  IRL  in 
order  to  report  a  value  for  the  SRL.  It  was  also  designed  to  be  useable  with  multiple  systems  architecting  tools  with 
minimal  modifications. 
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C.9.2  Application  toRT-34:  RT-27 

Where:  The  MPTs  developed  in  this  research  are  most  applicable  to  the  use  of  mature  technology  under  the  product 
aspect  of  level  1.  This  research  combined  two  pre-existing  MPTs  to  measure  the  readiness  of  software  systems  and 
combined  them,  creating  a  method  of  determining  the  overall  readiness  of  a  software  system  quantitatively. 

When:  SRL  levels  of  existing  systems  and  technologies  should  be  looked  at  during  the  requirements  and  concepts 
engineering  phases  of  an  SE  project.  This  way,  it  can  be  determined  if  a  system  is  adequate  for  integration  before  the 
design  process  starts,  or  it  can  be  deemed  unusable  and  a  different  system  can  be  chosen  for  analysis,  rather  than 
discovering  half  way  through  the  design  process  that  a  system  is  not  fit  for  integration  and  having  to  start  over  from 
scratch. 

How:  Using  the  SRL  quantifies  how  well  technologies  and  systems  can  be  integrated  together.  This  negates  the  need 
for  guesswork  or  qualitative  analysis  in  favor  of  more  reliable,  unbiased  data.  Using  this,  several  technologies  can  be 
compared  to  determine  which  is  a  better  fit,  or  one  existing  technology  can  be  looked  at  to  see  if  it  is  worth  the 
potential  risk  to  the  total  system  to  attempt  to  integrate  it,  or  if  it  would  be  more  beneficial  to  spend  the  time  to 
develop  a  better  fitting  technology  to  lower  the  risk. 


C.10  SERC  Research  Task  -  RT  30/31 


C.10.1  Description:  Graphical  CONOPS 

Overview:  Create  a  3D,  virtual  environment  where  stakeholders  and  concept  engineers  can  collaborate  to  create  and 
test  design  concepts  in  real  time.  Scenarios  can  be  run  with  these  models,  receiving  instant  feedback  and  data  so 
that  modifications  can  be  made  and  then  re-tested. 

Need:  Textual  CONOPS  can  take  months  to  make  and  may  not  involve  the  customer.  They  aren't  interactive  and  are 
incapable  of  running  "what  if"  scenarios.  Graphical  CONOPS  allow  multiple  stakeholders  to  collaborate  with  concept 
engineers,  which  may  allow  better  communication  of  needs  and  requirements  as  well  as  better  insights  into  the 
operating  systems. 

3D  gaming  has  been  shown  to  increase  productivity  by  channeling  intuition.  It  makes  us  happier  and  more  creative. 
When  faced  with  the  task  of  modeling  a  protein-cutting  enzyme,  3D  gamers  were  able  to  come  up  with  a  structure  in 
three  weeks,  whereas  scientists  had  been  struggling  with  a  solution  for  over  a  decade.  Similar  productivity  and 
efficiency  may  be  possible  with  Graphical  CONOPS. 

Concept:  The  creation  of  the  Graphical  CONOP  would  be  collaborative  between  two  people  or  groups  of  people: 
Primitive  Developer 

•  Highly  technical 

•  Programming  skills 

•  Technical  assistance  during  primitive  development 
CONOPS  Author 

•  Not  expected  to  have  programming  experience 

•  Knowledge  of  domains  and  subject  matter 
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Primitives  are  3D  models  with  one  or  more  domains  or  attributes.  Can  be  immature  with  a  3D  model,  or  a  domain  or 
both  but  no  specified  attributes,  or  mature  with  all  attributes  specified,  potentially  with  extra  attributes. 

CONOPS  Author  is  responsible  for  choosing  and  linking  primitives  to  create  a  system.  The  Primitive  Developer 
imports  objects,  creates  primitives  and  assigns  attributes  to  the  primitives. 


C.10.2  Application  to  RT-34:  RT-30/31 

Where:  Graphical  CONOPS  are  most  applicable  for  requirements  development,  under  the  process  aspect  of  level  2. 
Due  to  the  collaborative  nature  amongst  the  stakeholders  and  engineers,  it  could  also  be  conducive  to  the  intense 
knowledge  sharing  aspect  of  level  2.  Also,  due  to  the  digital  platform  that  the  CONOPS  are  developed  on,  it  can 
relate  to  the  digital  enablement  mentioned  that  facilitate  level  2. 

How:  Stakeholders  work  with  programmers  to  develop  models  of  a  system  they  believe  will  meet  their  needs  and 
requirements.  Programmers  are  then  able  to  execute  different  scenarios  using  these  models  with  real  time  results 
and  feedback  that  can  be  analyzed  to  update  the  model,  after  which  more  scenarios  and  tests  can  be  run.  This  way, 
stakeholders  can  be  sure  that  their  needs  are  being  met,  and  engineers  can  have  real  data  with  which  to  make 
necessary  changes  to  improve  their  initial  designs  faster,  shortening  the  development  time  significantly. 

When:  CONOPS  in  general  are  developed  in  the  beginning  of  a  project,  and  the  Graphical  version  is  no  different.  This 
is  the  initial  stage  of  development  where  the  stakeholder  requirements  are  synthesized  into  a  system  to  be  created. 


C.ll  SERC  Research  Task  -  RT-35 


C.ll.l  Description:  Kanban  in  Systems  Engineering 

Overview:  A  Kanban  (signal  card)  is  a  method  of  on-demand  scheduling  that  provides  a  visual  means  to  manage  flow 
within  a  process.  Rather  than  tasks  being  "pushed"  by  a  deadline,  tasks  are  "pulled"  based  on  availability  of 
resources  (signal  cards)  to  complete  that  task.  Each  signal  card  is  created  with  an  agreed  capacity  for  work,  and  one 
card  is  associated  with  each  piece  of  work.  Work  is  assigned  to  cards  based  on  determined  value  and  once  all  cards 
are  in  use,  no  further  work  is  assigned  until  more  cards  are  freed  up.  The  purpose  of  this  research  is  to  determine 
the  applicability  of  Kanban  scheduling  to  systems  and  software  engineering  in  a  rapid  response  environment. 

Goals:  The  overall  goal  of  this  research  is  to  verify  if  using  Kanbans  to  organize  projects  results  in  better  project 
performance.  Project  performance  is  measured  through  a  value  function  and  is  seen  as  achieving  value  along  one  or 
more  of  the  three  following  scales: 

•  Shortest-time  to  useful-value 

•  Highest-value  for  a  given-time 

•  Lowest  variation  in  transit  time 

Value:  Using  an  on-demand  Kanban  method  of  scheduling  was  predicted  to  provide  value  in  five  different  ways: 

1.  More  effective  integration  and  use  of  scarce  systems  engineering  resources 

a.  Value  functions  can  be  tailored  to  provide  efficient  scheduling,  maximizing  value  provided  by 
resources 

b.  Can  help  determine  if  delaying  other  service  requests  is  warranted  based  on  what  is  currently  being 
executed 


Contract  Number:  H98230-08-D-0171 


Page  128 

Report  No.  SERC- 2012-TR- 034 


TO  0015,  RT034 


UNCLASSIFIED 


2.  Flexibility  and  Predictability 

a.  Operates  with  shorter  planning  horizons,  increasing  flexibility 

b.  Provides  predictability  despite  dynamic  planning 

3.  Visibility  and  coordination  across  multiple  projects 

a.  Synchronize  activities  across  mutually  dependent  teams  by  coordinating  their  activities  through 
changing  value  functions 

b.  Show  status  of  work-in-progress  and  queued/blocked  work 

c.  Decreases  impact  of  long  latency  dependency  between  work  items  by  not  starting  work  on  items 
that  require  other  items  to  be  complete 

4.  Low  governance  overhead 

a.  Doesn't  require  major  changes  in  way  work  is  accomplished  or  imply  specific  organizational 
structures 

b.  Can  be  implemented  for  individual  projects  and  evolve  into  more  effective  governance  as  better 
ways  to  obtain  value  are  discovered 

c.  Visual  representation  of  flow  makes  issues  easily  identifiable 

d.  Metrics  are  inherent  to  the  system,  clearly  identify  problems,  and  help  track  improvements 

5.  Increased  project  and  system  value  delivered  earlier 

a.  Limits  work  in  progress  to  integrate  systems  and  project  engineering 

b.  Provides  specific  project  and  system-wide  work  item  value  determination 

c.  Maintains  long-term  investment 


C.11.2  Application  to  RT-34:  RT-35 

Where:  The  Kanban  scheduling  system  relates  to  both  the  real-time  management  aspect  of  level  3,  and  the  intense 
knowledge  sharing  aspect  of  level  2.  Kanban  also  enables  scheduling  flexibility,  by  allowing  changes  to  be  made  to 
how  work  is  done  if  the  requirements  change,  or  if  tasks  end  up  being  more  work  intensive  than  originally  expected 
The  continuous  integration  also  allows  for  complete  project  visibility  amongst  the  entire  team,  and  ensures  that  all 
the  separate  parts  work  together  as  they  are  supposed  to. 

When:  As  it  is  a  scheduling  method,  Kanban  is  best  implemented  after  the  requirements  and  concept  development 
part  of  a  design  project  are  complete  so  that  the  work  involved  to  complete  a  project  is  known.  One  of  the  key 
aspects  of  Kanban,  however,  is  that  tasks  and  resources  may  be  modified  throughout  the  lifecycle  of  the  project  to 
fit  changing  or  emerging  requirements,  or  adapt  to  unforeseen  circumstances. 

How:  One  of  the  points  made  in  this  research  was  that  it  didn't  require  much  change  to  the  way  a  team 
accomplishes  work  in  order  to  implement  a  Kanban  scheduling  system.  It  is  recommended  that  an  organization  use 
this  system  for  individual  projects  at  first,  and  allow  it  to  evolve  into  a  more  effective  governance  as  they  become 
more  comfortable  with  it,  and  as  they  begin  to  understand  better  ways  to  obtain  value  from  it. 
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Appendix  D.  Recommendations  for  Digitally-Enabled  Rapid  Acquisition 


There  are  practices  that  the  commercial  world  is  increasingly  reliant  on,  but  were  not  necessarily  seen  in  the 
government  rapid  organizations  we  visited.  We  may  not  have  seen  these  digitally-enabled,  information  technology 
centric  rapid  practices  because  we  didn't  explicitly  ask  questions  about  them  or  because  the  use  of  some  technology 
and  tools  is  limited  due  to  government  security  concerns,  restrictions  or  classification.  Below,  we  present  some 
examples  of  digital  enablement  that  we  have  observed  in  commercial  firms  that  we  suggest  have  the  potential  to 
facilitate  the  characteristics  described  in  Group  3,  such  as  more  intensive  knowledge  sharing  and  rapid  decision 
making  in  DoD  acquisition. 


Observation  D.l:  In  Commercial  Enterprises,  a  Priority  is  Given  to  Sharing  Quickly  Consumable  Cross-Data-Source 
Information 

In  the  commercial  world,  data  are  increasingly  shared  automatically  to  keep  up  with  the  need  for  rapid  decision¬ 
making.  Additionally,  because  rapid  decision-making  often  requires  the  simultaneous  consideration  of  different 
people,  process,  and  product  elements  of  the  project,  often  information  that  is  drawn  from  a  variety  of  different 
data  sources  need  to  be  shared.  However,  all  this  sharing  often  quickly  leads  to  information  overload.  Thus, 
commercial  rapid  projects  identify  ways  to  make  information  from  multiple  data  sources  not  only  available  but 
quickly  consumable. 

One  way  in  which  vast  amounts  of  information  is  shared  in  a  consumable  way  is  through  the  use  of  key  performance 
indicators  (KPIs)  and  thresholds  that,  when  exceeded,  trigger  a  message  sent  to  others.  For  example,  at  Western 
Digital,  KPIs  and  thresholds  for  in-process  manufacturing  scrap  are  used  to  signal  to  operations,  engineering,  and 
sourcing  personnel  when,  in  any  hour,  scrap  rates  are  higher  than  the  expected  threshold  (Houghton  et  al  2004). 

For  non-transactional  data,  such  as  customer  conversations  on  internet  discussion  boards,  KPIs  and  semantic 
analysis  tools  are  used  as  well  to  help  identify  customer  trends  and  then  to  automatically  alert  marketing  personnel 
of  the  opportunity  for  new  products  or  services  (Arthur  2012).  KPIs  can  be  used  with  keyword  searches  of 
customers'  use  of  social  media,  for  example,  to  alert  communications  personnel  of  the  need  for  rumor  or  customer 
damage  control  (Golla  2012).  The  application  of  KPIs  to  earlier  phases,  such  as  concept  development  are  also 
feasible.  KPIs  can  be  established,  for  example,  for  traceability,  requirements  definition,  and  risk-taking. 

Consummability  of  large  quantities  of  information  can  be  enhanced  in  ways  other  than  the  use  of  KPIs.  One  method 
is  to  devise  and  enforce  standards  around  the  use,  format,  and  access  of  many  different  data  sources  so  that  data 
from  the  different  sources  can  be  combined  to  lead  to  a  more  comprehensive  picture  of  a  topic  (Majchrzak  and 
Maloney  2008,  Mulholland  2006).  At  a  health  care  organization,  for  example,  data  from  a  patient's  past  Xrays,  blood 
tests,  prescriptions,  and  office  visits  have  been  electronically  standardized  so  that  emergency  room  doctors  can 
receive  the  information  they  need  immediately  upon  the  identification  of  an  emergency  room  patient,  resulting  in 
the  saving  of  lives.  Or,  when  an  engineer  needs  to  know  how  to  make  immediate  improvements  in  a  proposed 
manufacturing  process,  the  ability  to  access  and  combine  data  from  the  vendor,  past  manufacturing  output  of 
similar  products,  and  past  maintenance  records  can  help  the  engineer  to  quickly  answer  the  question.  While  our 
interviews  indicated  some  attempt  to  create  spreadsheets  that  allowed  for  different  engineers  to  quickly  do  trades 
analyses  (e.g.,  Aerospace's  CDC),  and  some  use  of  KPIs  underlying  project  management  dashboards,  we  found  little 
effort  to  create  standardized  databases  to  support  cross-source  consummability. 
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Observation  D.2:  In  Commercial  Enterprises,  Social  Networks  are  Used,  in  Real-Time,  to  Find  and  Share  Information. 

In  commercial  firms,  social  media  tools  are  increasingly  used  to  alert  others  within  a  firm  of  the  need  for  help  in 
problem-solving  or  service  provision.  Sales  personnel  use  instant  message  tools  such  as  Chatter  to  inform  others  of 
customer  needs  (Vance  2011).  Twitter  is  used  internally  to  a  firm  to  solicit  problem-solving  help  by  brief  messages 
of  the  need  for  information  or  problem-solving  help  are  broadcast  to  employees,  typically  resulting  in  immediate 
response  (Duffy  2011). 

Individuals  in  commercial  firms  are  also  increasingly  using  firm-based  social  media  to  be  instantly  alerted  about  new 
knowledge.  At  IBM,  for  example,  employees  use  their  internal  Facebook-like  profiles  to  indicate  the  projects  they 
are  working  on,  including  the  posting  of  recent  reports  or  listing  of  books  they  find  interesting  to  their  work 
(Majchrzak  et  al  2009).  Other  employees  "follow"  these  employees  so  that  they  are  informed  about  the  new 
documents  as  well.  At  Novell,  a  wiki  about  competitors  exist  (Wagner  &  Majchrzak  2006,2007)  to  which  many 
employees  subscribe.  As  employees  attend  conferences,  participate  in  online  forums,  or  become  aware  of 
competitors'  actions  in  other  ways,  the  employees  post  their  observations  to  the  wiki,  instantly  updating  the 
hundreds  of  other  employees  who  have  subscribed  to  the  wiki. 

While,  our  interviews  yielded  a  small  amount  of  use  of  instant  messaging  for  rapid  communication,  we  found  no  use 
of  social  networks  for  informing  rapid  personnel. 


Observation  D.3:  Use  of  "Living  Team  Toom"  Technologies  and  Processes  Enable  Successful  Programs 

There  has  been  significant  research  on  the  new  forms  of  working  that  virtual  teaming  demands  (e.g.,  Majchrzak  and 
Malhotra  2003,  Majchrzak  et  al  2004,  Malhotra  and  Majchrzak  2004,  2005,  Malhotra  et  al  2001,  2007).  A  living 
team  room  concept  is  one  that  replaces  email  by  people  altering  their  work  processes  so  they  communicate  by 
posting  draft  ideas  into  a  single  workspace,  that  also  has  version  control,  searching  and  chat  capabilities,  templates 
for  meeting  minutes,  ability  to  comment  on  others'  draft  ideas,  and  a  repository  to  house  finished  documents  are 
integrated  into  a  single  workspace  and  a  redesigned  work  process.  Their  redesigned  work  process  leverages  the 
unique  advantages  provided  by  asynchronous  and  synchronous  conversations:  with  synchronous,  team  members 
have  the  opportunity  to  engage  in  rapid  back  and  forth  dialogue  to  come  to  a  common  understanding  of  a  problem 
or  resolution;  with  asynchronous,  team  members  have  the  opportunity  to  think  more  deeply  about  a  problem  and 
offer  creative  solutions. 

The  redesigned  work  process  also  leverages  the  opportunity  provided  by  having  "living"  documents  in  a  central 
workspace  (such  as  wikis  or  modifiable  drawings).  As  such,  team  members  can  see  what  others'  are  drafting  and 
offer  early  advice  before  the  idea  becomes  so  well-developed  that  reluctance  to  make  changes  arises.  Moreover, 
by  commenting  and  rewriting  each  others'  work,  team  members  are  able  to  more  effectively  co-create  as  they  see 
each  other's  changes.  Having  the  evolution  of  the  team's  common  work  documented  in  real-time  helps  keep 
members  up-to-speed  with  each  other's  work  and  the  group's  current  status,  even  as  people  temporarily  leave  the 
team  for  vacations,  tours  of  duty,  working  on  others  projects,  etc. (Majchrzak  and  More  2012).  Finally,  living  team 
rooms  have  the  advantage  of  providing  information  about  the  status  of  the  team  to  experts  or  consultants,  allowing 
them  to  get  up  to  speed  rapidly. 

While  we  did  observe  some  use  of  Sharepoint  in  this  rapid  organizations  and  one  that  was  experiments  with  a  chat 
room,  the  uses  were  restricted  to  information  repositories,  rather  than  that  of  a  living  workspace. 
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Observation  D.4:  Internal  Crowdsourcing  can  Rapidly  get  Creative  Answers  to  Complex  Problems. 


Simply  because  a  project  team  is  working  on  a  project 
doesn't  mean  that  they  have  the  complete  knowledge 
needed  to  generate  the  most  creative  solutions  around 
difficult  complex  problems.  To  be  rapid,  companies  are 
increasingly  moving  toward  internal  crowdsourcing,  in 
which  they  announce  "challenges",  and  then  ask  people  to 
contribute  their  ideas  (Chesbrough  2006).  Recently,  these 
challenges  are  becoming  more  collaborative,  so  that  ideas 
are  not  just  being  contributed,  but  are  co-created  into 
more  complete  solutions.  Challenges  can  be  announced 
for  very  short  timeframes,  and  people  external  to  the  team  can  become  invested  in  helping  to  resolve  the  problems. 
In  the  past,  posting  challenge  problems  was  interpreted  as  a  negative  reflection  on  the  team's  inability  to  solve  its 
own  problems.  More  recently,  posting  challenge  problems  are  being  perceived  as  a  way  to  more  rapidly  resolve 
problems  with  fewer  internal  resources,  a  more  positive  and  rapid  way  to  solve  problems  [Chesbrough  2006], 


DARPA  Crowd  Sourcing  Demonstration 


•  2004  DARPA  Balloon  experiment! 

•  10  Weather  Balloons  released  across  US 

•  First  group  to  find  all  balloons  wins  $40,000  cash  prize 

•  MIT  found  balloons  in  9  hours 

•  Social  Networking  Technologies 

•  Sliding  scale  with  monetary  incentives  to  find  balloons 

•  Monetary  rewards  for  inviting  person  to  find  balloons 


Observation  D.5:  Integrative  tools  can  be  used  for  all  three  elements  of  rapid  (People,  Process,  Product)  in  the  areas 
of:  Estimation  of  Cost  and  Schedule,  Organizational  Fit. 

To  accelerate  the  system  development,  the  program  should  be  able  to  estimate  or  assess  their  status  and  find  out 
where  they  stand.  Two  tools  that  have  been  selected  to  highlight  in  relation  to  RT-34  are  TOP  Modeler  and 
Constructive  Rapid  Acquisition  Development  Model  (CORADMO).  For  each  tool,  a  description  is  provided  as  well  as 
a  context  for  when,  why,  and  how  the  tool  could  be  applied  to  expedited  development.  TOP  Modeler  is  described 
herein.  An  introduction  to  CORADMO  is  in  this  appendix,  and  a  separate  paper  with  more  details  is  in  Appendix  E. 


TOP  Modeler 

The  Technology  Organization  and  People  (TOP)  Modeler,  as  the  name  suggests  can  support  organizational  design 
and  analysis.  Using  a  knowledge-base  from  years  of  collected  data  and  research  literature,  TOP  Modeler  can  help  to 
optimize  (and  expedite)  an  organization's  system  development  processes.  This  can  help  examine  the  strong 
relations  to  the  observations  in  Group  1  and  identify  an  integrated  approach  to  people  (organization),  process  and 
product  (technology).  For  more  information, 

http://www-bcf.usc.edu/~maichrza/topmodeler/about-topmodeler.html 

What:  TOP  modeler  is  a  tool  developed  to  support  organization  analysis,  design,  and  reengineering  with  respect  to 
emergent  knowledge  processes.  TOP  stands  for  "Technology,  Organization,  and  People"  integration.  Structurally,  the 
TOP  Modeler  system  has  three  main  components:  a  knowledge-base  of  the  scientific  knowledge  about  organization 
design,  an  inference  engine,  and  an  interface  for  data  input  and  analysis  display. 

Why:  To  expedite  the  system  development  process,  the  TOP  Modeler  can  help  to  optimize  organizational 
capabilities  with  respect  to  knowledge  development,  capture,  management  and  application.  The  stakeholders  can 
use  TOP  Modeler  to  analyze,  design,  reengineer,  or  improve  program  /project/organization  status. 

When:  Anytime  in  the  project  lifecycle,  especially  at  the  beginning,  the  program  manager  or  responsible  person  can 
use  TOP  Modeler  to  evaluate  their  current  program/organization  status  or  evaluate  their  alternative  to-be  future 
skills,  processes,  and  tools. 
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How:  The  gap  for  improvement  can  be  assessed  when  the  stakeholder/user  describes  their  operation/business 
strategies.  These  current  strategies  are  compared  with  best  practices  and  the  TOP  Modeler  informs  the  users  of 
prioritized  gaps  that  need  to  be  mitigated.  The  gaps  are  described  in  terms  of  a)  3  types  of  operation/  business 
strategies,  which  are  Business  Objectives,  Process  Variance  Control  Strategies,  and  Organizational  Values,  and  b)  11 
enterprise  features,  which  are  Information  Resources,  Production  Process  Characteristics,  Empowerment 
Characteristics,  Employee  Values,  Customer  Involvement  Strategies,  Skills,  Reporting  Structure  Characteristics, 
Norms,  Activities,  General  Technology  Characteristics,  and  Performance  Measures  and  Rewards.  As  shown  in  figure 
D-l,  the  TOP  Modeler  Ferris  Wheel  shows  the  color-coded  thermometers  of  each  operation/business  strategies  and 
enterprise  features,  which  indicate  the  percentage  of  features  of  current  organization  status  matched  to 
benchmarked  best  practices  designate  by  the  user. 


Figure  D-l:  The  TOP  Modeler  Ferris  Wheel 

To  provide  a  quick  performance  status  update,  tradeoff  matrices  provide  immediate  feedback  to  stakeholders  on 
their  standpoint  (Figure  D-2).  If  the  current  organization's  operation/business  status  aligns  with  the  expected 
organization's  operation/business  strategy,  the  cell  will  be  shown  in  green;  otherwise  it  will  be  in  red.  This  feedback 
will  quickly  report  the  weak  aspects  and  encourages  users  to  improve  their  process  in  order  to  achieve  a  better 
project  performance. 
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Figure  D-2:  The  TOP  Modeler  -  Example  Tradeoff  Matrix 

Where:  The  TOP  Modeler  has  been  used  in  over  50  applications  of  organizational  redesign,  business  process 
redesign,  or  implementation  of  new  manufacturing  technology.  US  Air  Force,  ManTech  program,  the  National  Center 
for  Manufacturing  Sciences,  Digital  Equipment  Corporation,  Texas  Instruments,  Hewlett  Packard,  Hughes,  General 
Motors,  and  the  University  of  Southern  California. 

Foundation:  Through  extensive  literature  search  and  standard  setting  process  facilitated  by  National  Center  for 
Manufacturing  Sciences  (NCMS),  the  inference  engine,  the  knowledgebase  and  the  rules  were  derived  from 
sociotechnical  systems  theory  and  various  literature  reviews.  TOP  Modeler's  knowledgebase  was  developed  with 
the  support  from  the  U.S.  Air  Force  ManTech  program,  the  National  Center  for  Manufacturing  Sciences,  Digital 
Equipment  Corporation,  Texas  Instruments,  Hewlett  Packard,  Hughes,  General  Motors,  and  the  University  of 
Southern  California.  Information  about  the  development  of  TOP  Modeler  can  be  found  in  Markus  et  al  (2002). 

We  researched  this  tool  as  a  potential  candidate  for  assessing  inter-dependencies  of  critical  success  factors. 

However,  the  tool  required  significant  "modernization"  to  apply  to  today's  software  environment  (such  as  the  latest 
version  of  Windows).  The  concept  remains  valid  if  future  research  requires  this  analysis. 


CORADMO 

What:  CORADMO  is  part  of  the  Constructive  Cost  Model  (COCOMO)  suite  of  software  and  systems  engineering  tools 
to  estimate  development  cost,  illustrated  in  Figure  D-3.  CORADMO  stands  for  Constructive  Rapid  Application 
Development  Mode,  with  RAD  referring  to  application  of  any  of  a  number  of  techniques  or  strategies  to  reduce 
software  development  cycle  time.  COCOMO  has  been  used  over  30  years  by  software  developers  worldwide. 
CORADMO  extension  has  been  recently  updated  to  better  reflex  more  recent  agile  and  expedited  processes. 

The  foundation  of  CORADO  is  the  COCOMOII  tool  which  uses  5  scale  drivers  and  17  cost  factors  that  characterize  the 
team,  processes,  tools,  and  software  product  characteristics  to  calculate  estimated  effort  and  schedule  for  the 
software  development  activities.  Again  note  the  strong  relations  to  Level  1  observations  which  integrate  people, 
process  and  product.  CORADMO  provides  the  assessment  of  potential  schedule  savings  due  expedited  processes. 
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CORADMO  has  5  multiplicative  cost  factors  (product,  process,  project,  team,  and  risk  acceptance)  as  illustrated  in 
Table  D-3.  Under  each  multiplicative  cost  factor,  there  are  sub  factors  that  identify  various  ways  to  expedite  system 
and  software  development. 


Dates  indicate  the  time  that 
the  first  paper  was  published 
for  the  model. 


Legend: 

Model  has  been  calibrated  with  historical  project  data 

Model  is  derived  from  calibrated  model 

Model  has  been  calibrated  with  expert  (Delphi)  data 


Figure  D-3:  COCOMO  Suite  of  Engineering  Estimation  Model 

Why:  To  understand  the  range  of  potential  of  schedule  savings  due  expedited  processes. 

When:  When  the  organization  wants  to  evaluate  alternative  processes  for  system  and/or  software  development. 

The  earlier  to  use  and  guide  the  process,  the  more  saving  can  be  realized. 

How:  The  CORADMO  tool  uses  user  ratings  of  the  CORADMO  parameters  described  in  Table  D-l  to  calculate  the 
estimated  schedule  savings.  More  information  can  be  found  in  the  following  case  studies. 

Where:  has  been  used  by  many  DoD  contractors  since  2005.  CORADMO  extension  has  been  recently  updated  to 
better  reflex  more  recent  agile  and  expedited  processes. 

Foundation:  CORADMO  is  an  extension  to  the  systems  engineering  model  (COSYSMO)  that  estimates  the  schedule 
saving  associated  with  expedited  &  agile  processes.  COSYSMO  uses  14  cost  parameters  to  characterize  the  team, 
processes,  tools,  and  system  product  characteristics  to  calculate  estimated  effort.  Recently,  the  schedule  algorithm 
has  been  developed  to  calculate  the  estimated  schedule  based  on  estimated  effort  and  to  provide  the  needed  inputs 
to  the  CORADMO  system  engineering  calculation.  COCOMOII,  COSYSMO,  and  COSYSMO  schedule  algorithm  are 
primarily  based  on  US  DOD  system  and  software  development  programs. 

Appendix  E  gives  a  perspective  of  a  specific  example. 
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To  better  understand  the  MMPTs  (Models,  Methods,  Processes,  Tools)  project  factors,  Table  D-l  shows  examples  of 
how  MMPT's  differ  between  typical,  traditional  and  Expedited/Lean/Agile  projects. 

Table  D-l:  Methods,  Processes,  Tools  for  Traditional  and  Expedited  System  /  Software  development 


Category 

Traditional  Development 

Expedited  /  Lean  /  Agile  Development 

Project  Tracking 

•  Earned  Value  Management  System 

•  Formal  meetings 

•  Burndown  /  Velocity  charts 

•  Stand-up  meetings 

•  Kanban 

Project  Planning 

•  Formal  Plans 

•  PERT/Gantt  Charts 

•  User  Stories 

•  Kanban/  Scrum 

•  Planning  Poker 

Configuration  Management 

•  Formal  Change  Control 

•  Source  Code  Control 

•  Feature  Tracking 

•  Shared  Repository 

•  Wiki 

Requirements  Management 

•  Formal  Change  Control  Board 

•  Contract  Changes 

•  Winbook  (Requirements  negotiation, 
management,  refinement) 

Quality  Assurance 

•  Formal  reviews 

•  Formal  action  items  tracking 

•  Test  plans/  procedures/  scripts 

•  Simulation 

•  Built-in  Test 

•  Peer  reviews  /  pair  programming 

•  Markups  /  informal  notes  /  immediate 
changes  notification 

•  Test-Driven  Development 

•  Automated  Testing  Tools 

•  Problem  Tracking 

Design 

•  Top  Down  Development 

•  Reuse/COTS 

•  Combination  of  innovation,  reuse/COTS, 
prototyping,  refactoring 

Modeling 

•  Formal  models  (e.g.  SysML,  DODAF 
CAD/CAM)  with  change  control 

•  3D  printing 

•  Informal  models,  sketches  for  extension  to 
platforms  during  innovation  /  prototyping 

Risk  assessment  &  Improvement 

•  Quantitative  risk  assessment  and 
mitigation 

•  Risk  Exposure  Chart 

Observation  D.6:  Many  other  Systems  Engineering  Methods,  Processes  and  Tools  (MPT)  are  being  researched  that 

WILL  SOON  MATURE  FOR  USE. 

It  became  clear  during  the  early  execution  of  this  Research  Task,  that  many  of  the  other  SERC  RTs  may  generally 
improve  Systems  Engineering  within  any  program,  with  a  side  effect  that  programs  may  be  more  effective,  efficient, 
and  ideally  quicker.  Appendix  C  reviews  several  SERC  RTs  and  describes  the  relationship  with  RT-34  observations 
and  findings. 
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Appendix  E:  A  Model  for  Estimating  Agile  Project  Schedule  Acceleration 


This  is  a  draft  conference  paper  that  includes  RT-34  collaborators  from  USC.  It  has  been  submitted  to  the 
"International  Conference  on  Software  and  Systems  Process"  (ICSSP)  2013,  to  be  held  May  18-19,  2013,  in  San 
Francisco,  CA.  It  further  explores  the  use  of  CORADMO  as  a  tool  for  estimating  agile  project  schedule  acceleration. 

A  Model  for  Estimating  Agile  Project  Schedule  Acceleration 


Dan  Ingold,  Barry  Boehm, 
Supannika  Koolmanojwong,  Jo  Ann  Lane 
Center  for  Systems  and  Software  Engineering 
University  of  Southern  California 
Los  Angeles,  USA 

{dingold,boehm,koolmano,jolane}@usc.edu 


Abstract—  Accelerating  development  schedules  is 
increasingly  important  in  a  competitive  world. 
Reduced  time-to-market  is  a  key  response  to 
competitive  threats  in  the  commercial  sphere,  and 
rapid  response  in  deploying  military  systems  may  save 
lives  in  a  geopolitical  environment  characterized  by 
rapidly  emerging  and  ever-changing  physical  threats. 
Agile/lean  development  methodologies  show  promise 
in  providing  the  desired  schedule  acceleration,  but  it 
can  be  difficult  for  planners  to  determine  the  effects  of 
these  factors  on  schedule  duration,  and  to  make 
appropriate  choices  to  optimize  project  performance. 
The  Constructive  Rapid  Application  Development 
Model  (CORADMO)  attempts  to  quantify  the  effects  of 
key  schedule  drivers,  and  thus  enable  planners  to 
estimate  the  relative  schedule  that  will  result  from 
varying  these  parameters. 

Index  Terms—  Agile,  CORADMO,  estimation,  lean, 
modeling,  rapid  development. 

Introduction 

Accelerating  development  schedules  is  increasingly 
important  in  a  competitive  world.  Reduced  time-to- 
market  is  a  key  response  to  competitive  threats  in  the 
commercial  sphere,  and  rapid  response  in  deploying 
military  systems  may  save  lives  and  deter  adversaries  in 
a  geopolitical  environment  characterized  by  rapidly 
emerging  and  ever-changing  physical  threats.  Agile/lean 
development  methodologies  show  promise  in  providing 
the  desired  schedule  acceleration,  within  certain 
problem  domains  and  organizational  characteristics  [1], 


However,  we  have  found  that  many  projects  experience 
slower  schedules  by  jumping  into  agile  methods  without 
awareness  of  their  pitfalls.  These  include  making 
easiest-first,  hard-to-refactor  architectural 
commitments,  choosing  unscalable  or  incompatible  off- 
the  shelf  products,  accepting  unsuitable  on-site 
customer  representatives,  teambuilding  insufficiently, 
or  assuming  low  personnel  turnover. 

The  Constructive  Rapid  Application  Development  Model 
(CORADMO)  attempts  to  quantify  both  the  positive  and 
the  negative  effects  of  key  schedule  drivers,  and  thus 
enable  planners  to  estimate  the  relative  schedule  that 
will  result  from  varying  these  parameters.  CORADMO  is 
a  derivative  of  the  revised  Constructive  Cost  Model 
(COCOMO  II)  [2],  which  was  calibrated  against  larger 
projects  that  were  typically  optimized  to  reduce  cost.  In 
contrast,  the  goal  of  projects  using  agile/lean 
techniques  is  often  to  compress  schedule.  Further, 
COCOMO  II  generates  unreasonably  high  duration 
estimates  for  projects  of  fewer  than  two  person-years 
of  effort,  and  does  not  explicitly  consider  rapid 
development  techniques. 

The  original  CORADMO  described  in  Chapter  5  of  [2] 
operated  as  a  post-processor  to  adjust  the  cost  and 
schedule  estimates  coming  from  the  standard  cost- 
optimized  COCOMO  II  estimates.  The  COCOMO  II 
schedule  model  estimates  the  project  duration  D  in 
calendar  months  as  3.67  times  roughly  the  cube  root  of 
the  estimated  effort  in  person  months  (PM).  Thus,  a  27 
PM  effort  would  result  in  an  estimated  duration  of  3.67 
*  3  =  11  calendar  months,  and  an  average  staffing  level 
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of  27/11  =  2.45  people.  Such  a  small  team  minimizes 
communications  overhead  and  optimizes  effort,  but  11 
months  is  excessively  long  for  a  competitive  or  a  much- 
needed  product. 

As  we  were  calibrating  COCOMO  II,  we  were  also  seeing 
time-competitive  early-agile  projects  completing  27-PM 
projects  in  5  months  by  putting  an  average  of  5.4 
people  on  the  project.  In  some  well-jelled,  domain- 
experienced  Rapid  Application  Development  (RAD) 
organizations,  they  could  often  put  9  people  on  a  27-PM 
project  and  finish  in  3  months. 


This  motivated  the  development  of  CORADMO.  Its 
COCOMO  II  post-processor  used  a  nominal  square-root 
relationship  between  PM  and  D,  completing  a  27-PM 
project  in  5.2  months  with  an  average  team  size  of  5.2 
people.  It  then  adjusted  the  nominal  schedule  and  the 
originally-estimated  effort  by  applying  some  schedule 
acceleration-deceleration  factors  such  as  component 
reuse,  asset  prepositioning,  process  streamlining, 
collaboration  technology,  early  architecture  and  risk 
resolution,  and  RAD  personnel-team  capability. 

The  effort  and  schedule  multipliers  for  these  factors 


TABLE  I.  Schedule  Accelerators  and  Rating  Factors 


Accelerators/Ratings 

Very  Low 

Low 

Nominal 

High 

Very  High 

Extra  High 

Product  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

Extremely 

complex 

Highly 

complex 

Mod.  complex 

Moderately 

simple 

Highly  simple 

Extremely 

simple 

Element  Reuse 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate 

(50%) 

Considerate 

(70%) 

Extensive 

(90%) 

Low-Priority 

Deferrals 

Never 

Rarely 

Sometimes 

Often 

Usually 

Anytime 

Models  vs  Documents 

None  (0%) 

Minimal  (15%) 

Some  (30%) 

Moderate 

(50%) 

Considerate 

(70%) 

Extensive 

(90%) 

Key  Technology 
Maturity 

>0  TRL  1,2  or 
>1  TRL 3 

1  TRL  3  or  >  1 
TRL  4 

1  TRL  4  or  >  2 
TRL  5 

1-2  TRL  5  or 
>2  TRL  6 

1-2  TRL  6 

All  >  TRL  7 

Process  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent 

Operational  Concept, 
Requirements, 
Architecture,  V&V 

Highly 

sequential 

Mostly 

sequential 

2  artifacts 
mostly 
concurrent 

3  artifacts 
mostly 
concurrent 

All  artifacts 
mostly 
concurrent 

Fully 

concurrent 

Process  Streamlining 

Heavily 

bureaucratic 

Largely 

bureaucratic 

Conservative 

bureaucratic 

Moderate 

streamline 

Mostly 

streamlined 

Fully 

streamlined 

General  SE  tool 
support  CIM 
(Coverage, 

Integration,  Maturity) 

Simple  tools, 
weak 

integration 

Minimal  CIM 

Some  CIM 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

Project  Factors 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of 
personnel) 

Over  300 

Over  100 

Over  30 

Over  10 

Over  3 

<3 

Collaboration  support 

Globally 
distributed 
weak  comm.  , 
data  sharing 

Nationally 
distributed, 
some  sharing 

Regionally 

distributed, 

moderate 

sharing 

Metro-area 
distributed, 
good  sharing 

Simple 
campus, 
strong  sharing 

Largely 
collocated, 
Very  strong 
sharing 

Single-domain 

MMPTs  (Models, 
Methods,  Processes, 
Tools) 

Simple 

MMPTs, 

weak 

integration 

Minimal  CIM 

Some  CIM 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

Multi-domain 

MMPTs 

Simple;  weak 
integration 

Minimal  CIM 

Some  CIM  or 
not  needed 

Moderate  CIM 

Considerable 

CIM 

Extensive  CIM 

People  Factors 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs 
(Knowledge,  Skills, 
Agility) 

Weak  KSAs 

Some  KSAs 

Moderate 

KSAs 

Good  KSAs 

Strong  KSAs 

Very  strong 
KSAs 

Single-Domain  KSAs 

Weak 

Some 

Moderate 

Good 

Strong 

Very  strong 

Multi-Domain  KSAs 

Weak 

Some 

Moderate  or 
not  needed 

Good 

Strong 

Very  strong 

Team  Compatibility 

Very  difficult 
interactions 

Some  difficult 
interactions 

Basically 

cooperative 

interactions 

Largely 

cooperative 

Highly 

cooperative 

Seamless 

interactions 

Risk  Acceptance  Factor 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

Highly  risk- 
averse 

Partly  risk- 
averse 

Balanced  risk 
aversion, 
acceptance 

Moderately 

risk-accepting 

Considerably 

risk-accepting 

Strongly  risk- 
accepting 

Contract  Number:  H98230-08-D-0171 


Page  138 

Report  No.  SERC-2012-TR-034 


TO  0015,  RT034 


UNCLASSIFIED 


were  determined  such  that  a  well-jelled,  domain- 
experienced  RAD  project  would  be  estimated  as  9 
people  on  a  27-PM  project  for  3  months,  but  that  a 
misguided  RAD  project  would  take  more  like  40  PM  and 
9  months.  Unfortunately,  in  the  pre-2000  time  frame, 
we  did  not  have  a  critical  mass  of  data  to  calibrate  such 
a  model.  Recently,  though,  we  have  been  participating 
in  some  research  on  expediting  systems  and  software 
engineering  via  lean  and  agile  methods,  that  led  to  a 
expanded  set  of  product,  process,  project,  people,  and 
risk  factors  that  account  for  relative  schedule 
acceleration  and  deceleration  [3],  These  looked  like  a 
better  basis  for  developing  a  revised  CORADMO  set  of 
schedule  drivers  and  rating  scales,  and  are  discussed 
next. 

Method 

The  Systems  Engineering  Research  Center  (SERC) 
Research  Task  RT-34,  “Expedited  Systems  Engineering 
for  Rapid  Capability  and  Urgent  Needs,"  was  tasked  to 
study  ways  that  systems  engineering  might  be 
expedited,  particularly  within  the  aerospace/defense 
community.  Through  industry  and  government  contacts, 
the  study  identified  candidate  firms  and  agencies  that 
had  a  history  of  successfully  compressing  the 
development  time  of  projects.  In  a  series  of  onsite  visits 
and  in-depth  follow-up  interviews,  the  study  identified  a 
set  of  key  factors  [4]  that,  in  combination  with  factors 
derived  in  the  earlier  CORADMO  research  [2],  could  be 
used  to  model  RAD  projects'  schedule  acceleration 
(Table  I).  These  factors  fall  in  the  categories  of  product, 
process,  project,  people,  and  risk. 

Product  factors  describe  the  nature  of  the  system  to  be 
developed  across  five  sub-factors:  simplicity,  ability  to 
reuse  existing  elements,  ability  to  defer  lower-priority 
requirements,  degree  that  models  (prototypes, 
simulations,  etc.)  can  be  substituted  for  written 
documentation,  and  maturity  of  the  component 
technologies. 

Process  factors  characterize  the  development 
methodology  using  three  sub-factors:  concurrency  of 
artifact  development  (operational  concept, 
requirements,  code,  etc.);  degree  of  process 
streamlining;  and  the  coverage,  integration,  and 
maturity  (CIM)  of  tools  used  to  support  the 
development  process  [5], 

Project  factors  span  four  sub-factors  describing 
execution  of  the  development  effort:  project  staff  size; 


degree  and  nature  of  team  collaboration;  CIM  of  the 
single-domain  models,  methods,  processes,  and  tools 
(MMPTs)  employed;  and  CIM  of  the  multi-domain 
MMPTs  used,  where  required. 

People  factors  describe  the  project  staff  using  four  sub¬ 
factors:  general  knowledge,  skills,  and  agility  (or,  ability 
to  thrive  with  the  more  chaotic  nature  of  the  agile/lean 
process)  [1];  KSAs  specific  to  the  primary  problem 
domain;  KSAs  spanning  multiple  problem  domains, 
where  needed;  and  team  compatibility  [6], 

Finally,  the  Risk  factor  characterizes  the  project 
stakeholders'  willingness  to  accept  rapid  but  imperfect 
solutions  [1],  Stakeholders  may  range  from  highly  risk- 
averse,  to  strongly  risk-accepting. 

As  discussed  in  the  Introduction,  a  good  baseline 
estimate  for  schedule  reduction  through  rapid 
development  methods  has  been  found  to  be 
proportional  to  the  square  root  of  effort.  CORADMO 
estimates  duration  (D)  as  the  product  of  multipliers 
associated  with  the  factors  described  above  (F,)  and  the 
square  root  of  baseline  effort  in  person-months  (PM), 
also  has  multipliers  that  adjust  the  original  effort 
estimate  to  reflect  the  effect  of  RAD  on  effort. 

D  =  [I  Fi  sfPM. 

As  seen  in  Table  I,  each  of  the  proposed  factors  is  rated 
along  a  6-value  Likert  scale  ranging  from  Very  Low  to 
Extra  High,  where  factors  rating  lower  in  the  scale  tend 
to  extend  the  schedule,  and  those  rating  higher  to 
reduce  it.  Initial  values  of  the  schedule  acceleration 
multipliers  were  chosen  to  span  a  relatively  small  range 
of  duration  expansion  and  reduction,  pending  model 
calibration.  Our  evaluation  of  rapid  development 
projects  in  this  research,  however,  suggests  that  people 
factors  [4]  and  risk  tolerance  [1]— which  tracks 
willingness  to  accept  some  product  imperfections  to 
improve  schedule— have  greater  effects  than  the  other 
factors,  which  is  reflected  in  the  greater  span  of  their 
associated  schedule  multipliers. 

We  evaluated  the  CORADMO  model  against  a  12- 
project  dataset  of  diverse  but  single-company  projects 
executed  by  a  Midwest  software  development  firm  that 
used  agile  practices,  and  that  supplemented  those 
practices  with  systems  engineering  (SE)  processes 
distinguishing  their  approach  from  typical 
BigDesignUpFront-avoiding  agile  projects.  These  SE 
practices  included  detailed  business  process  analyses, 
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Delphi  estimates  of  software  testing  effort,  risk-based 
situation  audits,  and  componentized  architectures, 
among  others.  Use  of  systematic  SE  processes  by  the 
firm  was  considered  to  make  these  projects  more 
comparable  to  the  SE  practices  applied  in  the  more 
complex  aerospace/defense  projects  from  which  the 
factors  in  Table  I  were  derived. 

The  model  was  also  applied  to  a  case  study  derived 
from  observations  of  aerospace  and  commercial  firms 
that  have  been  affiliated  with  the  Center  for  System  and 
Software  Engineering  (CSSE)  at  the  University  of 
Southern  California  (USC).  While  this  case  study  is  not 
directly  traceable  to  any  single  firm,  it  is  representative 
of  the  range  of  projects  and  capabilities  that  we  have 
seen  in  real  firms.  This  application  of  the  model  allowed 
us  to  characterize  the  types  of  schedule  effects  that  one 
might  expect  to  see  by  varying  the  factors,  which  we 
plan  to  validate  against  actual  projects  in  future 
research. 

Results 


Calibration  to  Commercial  Rapid  Development  Projects 

Table  II  presents  a  dataset  of  twelve  commercial  rapid 
development  projects  ranging  in  size  from  10  KLOC 
(thousands  of  source  lines  of  code)  to  400  KLOC,  of 
varying  complexity  and  technology.  We  rated  these 
projects  against  the  Product,  Process,  Project,  People 
and  Risk  factors  discussed  above  to  compute  the 


product  of  the  schedule  acceleration  factors,  and  to 
compare  them  against  the  D/i/PM  calculated  from  the 
reported  project  duration  and  effort. 

Factor  ratings  were  selected  based  upon  the  reported 
characteristics  of  each  project,  and  of  the  firm  as  a 
whole.  The  projects  that  employed  C++  technologies 
received  Low  (L)  Product  Simplicity  ratings  as  compared 
with  the  other  HTML-Visual  Basic  projects  and  the 
described  product  complexity;  the  "Hybrid  Web/Client 
Server"  Product  was  rated  Low  (L)  due  to  its  high  degree 
of  innovation  and  requirements  churn.  For  the  Process 
factor,  most  projects  used  a  highly  concurrent 
development  process,  resulting  in  a  Very  High  (VH) 
rating;  some  projects  reported  using  more  complex 
mixes  of  technology  that  suggest  less  concurrency,  and 
therefore  received  lower  ratings.  Reported  variation  in 
project  staff  sizes  is  the  primary  reason  for  the  varying 
Project  ratings.  The  staff  was  described  as  being  very 
capable  and  senior-level,  and  so  the  People  factor  rated 
at  Very  High  (VH)  across  the  board.  Similarly,  the  firm 
documented  a  consistent  and  rigorous  development 
approach,  balancing  good  engineering  against 
development  speed,  and  hence  were  all  rated  at 
Nominal  (N)  Risk  acceptance. 

The  product  of  the  selected  rating  factors  is  shown  in 
the  Multiplier  column  of  Table  II,  and  should  be 
compared  against  the  value  in  the  Duration/VPM 
column,  calculated  from  actual  duration  and  effort.  The 
close  correspondence  of  these  values  in  the  Error 


TABLE  II.  Commercial  Projects  Rating  Factors  and  Analysis 


Application  Type 

Technologies 

Person 

Months 

Duration 

(Months) 

Duration 

/  Vpm 

Product 

Process 

Project 

People 

Risk 

Multi¬ 

plier 

Error 

% 

Insurance  agency  system 

HTML/VB 

34.94 

3.82 

0.65 

VH 

VH 

XH 

VH 

N 

0.68 

5% 

Scientific/engineering 

C++ 

18.66 

3.72 

0.86 

L 

VH 

VH 

VH 

N 

0.80 

-7% 

Compliance  -  expert 

HTML/VB 

17.89 

3.36 

0.79 

VH 

VH 

XH 

VH 

N 

0.68 

-15% 

Barter  exchange 

SQL/VB/  HTML 

112.58 

9.54 

0.90 

VH 

H 

H 

VH 

N 

0.75 

-16% 

Options  exchange  site 

HTML/SQL 

13.94 

2.67 

0.72 

VH 

VH 

XH 

VH 

N 

0.68 

-5% 

Commercial  HMI 

C++ 

205.27 

13.81 

0.96 

L 

N 

N 

VH 

N 

0.93 

-3% 

Options  exchange  site 

HTML 

42.41 

4.48 

0.69 

VH 

VH 

XH 

VH 

N 

0.68 

-1% 

Time  and  billing 

C++/VB 

26.87 

4.80 

0.93 

L 

VH 

VH 

VH 

N 

0.80 

-14% 

Hybrid  Web/client-server 

VB/HTML 

70.93 

8.62 

1.02 

L 

N 

VH 

VH 

N 

0.87 

-15% 

ASP 

HTML/VB/SQL 

9.79 

1.39 

0.44 

VH 

VH 

XH 

VH 

N 

0.68 

53% 

On-line  billing/tracking 

VB/HTML 

17.20 

2.70 

0.65 

VH 

VH 

XH 

VH 

N 

0.68 

4% 

Palm  email  client 

C/HTML 

4.53 

1.45 

0.68 

N 

VH 

VH 

VH 

N 

0.76 

12% 
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column  suggests  that  the  acceleration-deceleration 
factors  are  appropriate,  although  additional  work 
remains  in  that  the  calculated  factors  suggest  greater 
schedule  acceleration  that  was  actually  observed.  The 
"ASP"  project  is  an  outlier  that  we  cannot  explain  from 
the  data  reported. 


Agile  SE  Adoption  Case  Study 

This  case  study  illustrates  the  use  of  the  revised 
CORADMO  model  in  explaining  the  differences  in 
schedule  acceleration  for  various  project  approaches. 
The  baseline  situation  for  the  case  study  is  a 
hypothetical  company  division  specializing  in 
performing  early-SE  activities  for  defense  applications  in 
a  diversified  company,  generally  involving  teams  of 
roughly  20  systems  engineers  (SEs).  The  division  has 
traditionally  applied  a  sequential  waterfall  or  "Vee" 
model  to  define  a  system's  operational  concept  and 
requirements,  and  then  developed  a  system 
architecture  that  satisfies  those  requirements.  Defense 
needs  for  rapid  response  projects,  however,  have  led 
the  division  to  desire  a  change  to  a  more  agile 
approach. 

The  baseline  "as-is"  factor  ratings  for  the  division  are 
shown  as  boxes  marked  "X"  in  Table  III.  The  additional 
sub-factor  detail  available  in  this  hypothetical  division 
case  study,  only  some  of  which  was  inferable  in  the 
commercial  data,  raises  a  question  of  how  sub-factors 
that  span  a  range  of  ratings  should  be  handled.  In 
COCOMO,  a  particular  rating  is  chosen  based  on  the 
preponderance  of  sub-factors  that  match  the  situation, 
possibly  modified  based  on  the  expert  judgment  of  the 
modeler.  Here  we  have  decided  to  average  the 
multipliers,  reasoning  that  higher  ratings  in  some  sub¬ 
factors  offset  lower  factors  in  others. 

The  division's  four  product  factor  ratings  are: 
moderately  complex  (N);  sufficiently  diverse  to  make 
reuse  infeasible  (VL);  non-subsettable  so  that  low- 
priority  deferrals  are  infeasible  (VL);  able  to  use  models 
vs.  documents  only  15%  of  the  time  (L);  and  involving 
only  one  or  two  slightly  immature  (TRL  6)  technology 
elements  (VH). 

The  three  process  factor  ratings  for  the  division  are: 
highly  sequential  SE  processes  (VL);  largely  bureaucratic 
internal  and  external  project  and  business  processes  (L); 
moderate  SE  tool  coverage,  integration,  and  maturity 
(H). 


TABLE  III.  As-Is  Rating  Factors 


Accelerators/Ratings 

VL 

L 

N 

H 

VH 

XII 

Product  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

X 

Element  Reuse 

X 

Low-Priority  Deferrals 

X 

Models  vs  Documents 

X 

Key  Technology 

Maturity 

X 

Process  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent  Operational 
Concept,  Requirements, 
Architecture,  V&V 

X 

Process  Streamlining 

X 

General  SE  tool  support 
CIM  (Coverage, 
Integration,  Maturity) 

X 

Project  Factors 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of 
personnel) 

X 

Collaboration  support 

X 

Single-domain  MMPTs 
(Models,  Methods, 
Processes,  Tools) 

X 

Multi-domain  MMPTs 

X 

People  Factors 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs 
(Knowledge,  Skills, 
Agility) 

X 

Single-Domain  KSAs 

X 

Multi-Domain  KSAs 

X 

Team  Compatibility 

X 

Risk  Acceptance  Factor 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

X 

The  division's  four  project  factors  are:  project  SE  staff 
size  between  10  and  30  people  (H);  good  collaboration 
support  across  several  metro-area  facilities  (H); 
moderate  CIM  for  single-domain  MMPTs  (H);  and 
minimal  CIM  for  multi-domain  MMPTs  (L). 

The  people  factor  ratings  are:  good  general  knowledge, 
skills,  and  agility  (KSAs)  (H);  good  single-domain  KSAs 
(H);  good  multiple-domain  KSAs  (H);  but  some  difficult 
team  interactions  (L). 

The  divisions  approach  to  risk  is  evenly  balanced 
between  risk-aversion  and  risk  acceptance,  leading  to  a 
nominal  rating  (N)  and  no  effect  on  the  schedule. 

The  selected  ratings  result  in  the  following  factor 
multiplier  values,  which  calculates  an  overall 
acceleration  factor  of  1.01,  suggesting  the  division's 
approach  will  result  in  a  schedule  duration  close  to  the 
nominal  case: 

•  Product:  1.0*1.09*1.09*1.05*0.92=  1.15 

•  Process:  1.09*1.05*0,96=  1.10 

•  Project:  0.96*0.96*0.96*1.04  =  0.92 
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•  People:  0.94*0.94*1.06*0.94  =  0.88 

•  Risk:  1.0 

The  division  initially  attempts  a  change  to  a  more  agile 
process  approach  by  producing  multiple  artifacts 
(operational  concept,  requirements,  and  architecture) 
concurrently,  instead  of  sequentially,  as  shown  with  the 
green  arrow  in  Table  IV.  This  was  expected  to  reduce 
the  schedule  by  13%,  from  a  1.09  multiplier  to  0.96. 

However,  when  the  project  was  performed,  the 
organization  was  surprised  that  the  actual  schedule  was 
about  15%  longer  rather  than  shorter.  In  performing  a 
review  of  the  cause  of  this,  the  division  found  that  the 
project  focused  only  on  its  agile  and  concurrency 
aspects,  and  neglected  to  examine  the  potential  side 
effects  of  a  too-hasty  changeover. 

With  respect  to  the  other  CORADMO  factors,  the 
project  missed  several  other  factors  that  affect  the 
overall  schedule.  These  include  missed  opportunities  in 
addressing  some  of  the  improvable  SE  schedule 
influence  factors,  but  not  others,  such  as  the  largely 
bureaucratic  internal  and  external  project  and  business 
processes,  and  the  Low-rated  multi-domain  MMPTs  and 
KSAs.  Other  detrimental  effects  resulted  from  pitfalls  in 
transitioning  from  sequential,  heavyweight  processes  to 
agile  processes,  as  illustrated  by  the  red  arrows  in  Table 
IV: 

•  Key  Technology  Maturity.  In  producing  artifacts 
concurrently,  the  project  overlooked  some 
interactions  between  subsystems,  and 
mischaracterized  the  maturity  of  technologies 
through  insufficient  analysis.  This  resulted  in  a 
change  to  a  Nominal  from  a  Very  High  rating, 
causing  a  slowdown  factor  of  1.0/0.92=1.09 

•  General  SE  tool  support.  Using  a  mix  of  agile 
MMPTs  tools  and  traditional  MMPTs  made  their 
MMPTs  less  integrated,  increasing  the  sub-factor 
rating  to  High  from  Very  High,  for  a  slowdown 
factor  of  1.0/0.96=1.04. 

•  General  SE  KSAs.  Rapid  development  approaches 
required  a  different  mindset  from  team 
members,  causing  a  slowdown  factor  of 
1.0/0.94=1.06. 

•  Team  Compatibility.  A  different  style  of 
collaboration  is  often  necessary  in  agile 
development,  requiring  frequent  face-to-face 


discussions  rather  than  serialized  document 
reviews.  Team  members  or  management  may  be 
uncomfortable  with  or  hostile  to  such 
interactions,  resulting  in  an  increase  of  this  sub¬ 
factor  to  Nominal  from  High,  for  a  slowdown  of 
1.0/0.94=1.06. 


TABLE  IV.  Initial  To-be  Rating  Factors 


Accelerators/Ratings 

VL 

L 

N 

H 

VH 

XII 

Product  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

X 

Element  Reuse 

X 

Low-Priority  Deferrals 

X 

Models  vs  Documents 

X 

Key  Technology 

Maturity 

X 

Process  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent  Operational 
Concept,  Requirements, 
Architecture,  V&V 

►  X 

Process  Streamlining 

X 

General  SE  tool  support 
CIM  (Coverage, 
Integration,  Maturity) 

X  4 

Project  Factors 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of 
personnel) 

X 

Collaboration  support 

X 

Single-domain  MMPTs 
(Models,  Methods, 
Processes,  Tools) 

X 

Multi-domain  MMPTs 

X 

People  Factors 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs 
(Knowledge,  Skills, 
Agility) 

X 

fc 

Single-Domain  KSAs 

X 

Multi-Domain  KSAs 

X 

Team  Compatibility 

Risk  Acceptance  Factor 

1.13 

1.06 

1.0 

,1.94 

0.89 

0.84 

X 

Therefore,  although  one  of  the  process  sub-factors 
improves  as  a  result  of  the  division's  improvement 
initiative,  due  to  unintended  effects  several  other  sub¬ 
factors  become  worse.  The  resulting  CORADMO 
estimate  of  the  net  effect  is 

0.88*1.09*1.04*1.06*1.06=1.13.  Thus,  the  CORADMO 
factor  analysis  not  only  explained  their  slowdown  factor 
of  about  15%,  but  also  it  provided  them  with  a  roadmap 
of  further  agile  improvements  they  could  make  to  begin 
to  experience  agile  speedups,  along  with  estimates  of 
the  impact  that  these  would  have  on  their  schedule. 
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As  illustrated  here,  when  an  organization  is  considering 
an  improvement  initiative,  it  can  use  CORADMO  as  a 
tool  to  identify  potential  side  effects  and  analyze  their 
impacts.  With  this  information,  the  organization  can 
then  take  steps  to  ensure  these  side  effects  are 
countered  with  additional  aspects  of  the  improvement 
initiative,  and  thus  improve  the  likelihood  of  achieving  a 
goal  of  schedule  reduction. 

Reconsidering  its  improvement  initiative  given  this 
analysis,  Table  V  shows  the  company  preparing  for  the 
future  by  restoring  the  sub-factors  whose  ratings  had 
worsened  to  their  baseline  values  (the  yellow  arrows  in 
Table  V),  through  being  aware  of  the  potential  problems 
and  therefore  taking  positive  steps  to  avoid  them.  This 
would  eliminate  an  overall  slowdown  factor  of 
1.09*1.04*1.06*1.06=1.29.  They  also  added  further 
initiatives,  illustrated  with  green  arrows  in  Table  V,  to: 

•  Perform  concurrent  V&V  along  with  concurrent 
Operational  Concept,  Requirements,  and 
Architecture  activities,  raising  the  rating  to  Very 
High  from  High,  for  a  speedup  factor  of 
0.92/0.96=0.96 

•  Improve  bureaucratic  internal  and  external 
project  and  business  processes  to  be  at  least 
moderately  streamlined,  for  a  speedup  factor  of 
0.96/1.04  =  0.92. 

If  these  goals  were  achieved,  the  resulting  CORADMO 
multiplier  estimate  would  be 

1.13*0.96*0.92/1.29=0.77,  for  a  speedup  of  23%  over 
their  original  situation.  Further,  this  schedule  reduction 
would  be  achieved  through  improving  only  three 
process  sub-factors,  and  simply  remaining  at  the  pre¬ 
initiative  levels  for  all  other  factors,  through  being 
aware  of  potential  detrimental  effects  and  taking  steps 
to  ensure  they  would  not  occur. 

On  their  next  project,  they  were  not  able  to  realize  the 
full  23%  speedup,  but  were  able  to  realize  a  15% 
speedup  factor  instead  of  a  15%  slowdown  factor. 
Continuing  use  of  the  CORADMO-based  schedule 
acceleration  framework  enabled  them  to  not  only 
achieve  their  initial  23%  speedup  target,  but  to  identify 
additional  improvements  that  accelerated  their 
schedules  even  further. 


TABLE  V.  Final  To-be  Rating  Factors 


Accelerators/Ratings 

VL 

L 

N 

H 

VH 

XII 

Product  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Simplicity 

X 

Element  Reuse 

X 

Low-Priority  Deferrals 

X 

Models  vs  Documents 

X 

Key  Technology 

Maturity 

1 

_ 

\  X 

Process  Factors 

1.09 

1.05 

1.0 

0.96 

0.92 

0.87 

Concurrent  Operational 
Concept,  Requirements, 
Architecture,  V&V 

-1 

►  X 

Process  Streamlining 

►  x 

General  SE  tool  support 
CIM  (Coverage, 
Integration,  Maturity) 

r 

v 

Sx 

Project  Factors 

1.08 

1.04 

1.0 

0.96 

0.93 

0.9 

Project  size  (peak  #  of 
personnel) 

X 

Collaboration  support 

X 

Single-domain  MMPTs 
(Models,  Methods, 
Processes,  Tools) 

X 

Multi-domain  MMPTs 

X 

People  Factors 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

General  SE  KSAs 
(Knowledge,  Skills, 
Agility) 

V 

r 

Single-Domain  KSAs 

X 

Multi -Domain  KSAs 

X 

_ l\ 

Team  Compatibility 

X 

Risk  Acceptance  Factor 

1.13 

1.06 

1.0 

0.94 

0.89 

0.84 

X 

Discussion 

The  combination  of  the  original  CORADMO  model  and 
the  additional  insights  on  product,  process,  project, 
people,  and  risk  factors  provided  by  the  SERC  RT-34 
analyses  enabled  the  revised  CORADMO  model  to 
explain  the  variations  in  schedule  acceleration  among 
the  projects  in  Table  II.  This  is  encouraging,  but  it  is 
unknown  to  what  extent  the  model  will  accurately 
describe  projects  outside  this  limited  set.  We  are  in  the 
process  of  collecting  additional  data  points  over  a  wider 
variety  of  projects.  These  data  should  allow  us  to  better 
calibrate  the  model  and  evaluate  its  wider  applicability. 
At  a  minimum,  though,  the  model  can  be  used  as  a  good 
checklist  for  assessing  an  organization's  status  and 
prospects  with  respect  to  schedule  acceleration. 

Overall,  as  was  observed  in  [1],  an  organization's  culture 
is  one  of  the  critical  factors  in  its  ability  to  achieve  near- 
term  gains  from  going  to  agile  methods.  A  good  deal  of 
careful  re-culturing  is  needed  to  take  an  organization  of 
people  who  feel  comfortable  and  empowered  by  having 
sets  of  standard  policies,  practices,  and  procedures  that 
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define  success  in  the  organization,  to  an  organization 
where  people  feel  comfortable  and  empowered  when 
there  is  a  minimum  of  such  standard  policies,  practices, 
and  procedures.  Several  of  the  CORADMO  factors  can 
help  in  gauging  an  organization's  progress  in  making 
such  transitions. 
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